Back to General discussions forum
Perhaps "communization" is not the best term or may have unnecessary negative connotations - feel free to propose topic title improvement.
While our site is not in top-10 of the whole internet, it still have some audience and some content, in creation of which many people have participated. Some years ago I started thinking how best to preserve it in case of anything unexpected. Recently it was mentioned that the very least effort should be publishing the problem statements separately so they won't suddenly perish. Our colleague Clive contacted me expressing the thought that something more than "least effort" could be anticipated, like sharing or delegating administrative functions etc. I thought of such possibility often but failed to come to some good idea (mainly about how to do it logically/technically and who may be interested in being involved).
Now let me outline main points as I see them with the goal that you can review and perhaps suggest some additions or improvements.
What goals we may have? Firstly - that in the event of something happening to me or to the site, some initiative colleagues have access to materials needed to redeploy it somewhere else. Secondly - perhaps that some administrative duties, like accepting new tasks - could be done by some elect members.
As website consists mainly of two entities - source code and data in the database - how can they be shared (in order the site could be recreated without myself)? For the code most obvious answer seems to be "opensource it", while data has some things which should be kept relatively secretly (user authentication data, problem checking scripts) - so probably data backups should be made periodically and access to them should be given to few trusted persons. There are other auxiliary things (sandbox, server and code for interactive problems etc) - but let's ignore this for now just not to add complexity.
Opensourcing the site code is technically easy - just copying it from the private repository - main hampering factor is that there supposedly could be some security flaws and making them public brings some more risks. However there seem to be not much which could be stolen from the site in case of some breach exploited. I can't see remedy for this except careful and somewhat tedious review process. Thus probably it is going to happen in steps: firstly code is copied bit by bit into another private repository to which more people (but still limited number) have access with the goal of trying it, reviewing it and recommending any necessary improvements and removals. When the whole thing is less or more reviewed, that repo is open to public. Technically from this point any enterprise or university can fetch the code and deploy its own clone for own needs.
Management of tasks sounds perhaps more tricky - I can add some logic that chosen people can be given some "guru" rights - they can add new tasks, invisible, test the setter/checker code, refine statement and then make it visible. The main drawback is that currently here is zero "collaborative" features, so such "gurus" need to be careful not to override each one additions etc. Also it is a question whether actions of one guru need to be approved by other(s). Or on contrary whether one can prevent/cancel other's actions if anything went wrong.
Please feel free to discuss or express your wishes if something important is forgotten!
Hi Friend!
This is impressive, but I pray not to go too far without synchronising our intentions :)
I'm sorry for I failed to response in time - I thought some more people may join discussion and meanwhile tried some different direction (I'll share about it bit later).
Reviewing your points:
I lack frontend development skills
Me too, though of course existing scripts/styles and templates could be shared. Regretfully templates are also in PHP and if you stick to idea of doing this in Python you'll need to change them to jinja or something alike, as I understand.
Concluding During the past week or two watching the lack of excitement about this activity :) I gradually have come to slightly different / alternative approach which I hope to uncover in a few days. Please get tuned, there'll be something to lay our hands on and wrap our minds about!
Good morning Rodion,
I don’t think there is a lack of willingness to help, rather I feel we are uncertain how exactly we can help.
I am most certainly keen to contribute in some way, but my total lack of web development experience
somewhat limits direct practical support.
But there are different aspects:
This is impressive, but I pray not to go too far without synchronising our intentions :)
All right! I will wait until we have a clear understanding of our intentions.
I'd prefer not to investing our time on rewriting the site to different language.
I do not have the source code of original site, so I wasn't rewriting anything, just writing something from scratch. Of course, I understand the amount of work you have done, so if instead of reinventing the wheel I can help you with improving already written code, this would be much better.
By the way, thank you for mentioning OpenAbbey and linking the old site version. I will look into these, as it sound very interesting.
Management of tasks just by anyone having certificate sounds a bit dangerous.
No problem! We can then stick to current system, where programming problems are proposed through forum and reviewed by you or people you choose.
I want to echo everybody that I am willing to help, but personally I'm somewhat unsure of my capacity to do so. I've been playing around with PHP/CSS/mySQL on a free server using what I've learned here as a launchboard, but this website seems a good deal larger in scale!
Either way, I'm willing to help in whatever way I might be able :)
You see how many responses, Rodion. Even if we lack some necessary skills, we really can do something together.
Sorry people, haven't much success in CodeAbbey development this weekend...
But studying abbey-2 and MessCode (SourceForge variant of CodeAbbey) is a really educational way to spend one's time :)
Hi Friends, sorry for the delayed answer - yes now we see there are quite a "Mighty Handful" of colleagues - but as Mathias correctly pointed out it is bit difficult to understand what kind(s) of help are needed.
Not that I myself have quite clear idea yet. Let's look at the points raised by Mathias:
Support of day-to-day running of the site.
What "day-to-day" activities are necessary? Well, besides answering at forum (which already is done by you all very well) there are mainly two activities - adding new problems and handling too blatant cheaters. The second can be skipped for now - while how exactly to deal with the first - the simplest way is to allow certain users the access to add / edit problem interface. And create necessary improvements and instructions to it. Main concern is that has no features to simplify cooperative work. But we can invent them by and by.
Support of backup process.
Yes, probably it should be like the backups are created weekly, and for example, Graeme
and Clive
receive emails
notifying backup is successful - and where to find it (some link probably). If the site is down for prolonged time and
seemingly I'm unresponsive, backups are handed by them to some other members who will be willing to redeploy
everything somewhere else. Though it is an open question how to share the domain name.
Financial support - I know that is not always straightforward in these times.
True but mostly it's ok for now - the most annoying thing is generally that different things are managed in different places. For the main server (Hostgator in US) I was able to make bank transfer for 3 years ahead, for sandbox (which is also used for checker in some "special" problems) they (BlueVPS in Europe) luckily accept cryptopayments, while domain name itself currently is in hands of BigRock (which is Indian). Simply explaining some of my foreign friends "login here, pay there" with clumsy interfaces of all these facilities makes more headache than the issue of spending money themselves :) It would be great probably to unite managing of all those first under some single service. We'll see.
Maybe also sharing of all login details with one or two trusted parties as a further safety net.
That is possible, just regretfully may be futile (if we speak of credentials to hosting, rather than administrative credentials of the site itself) - nowadays most logins from unusual location may require confirmation by email and logging in to email may require confirmation by phone. Additionally current hosting is used for some other of my small projects so it is somewhat messy. So probably mirroring and gradually shifting it to some carefully chosen hosting which allows properly sharing access could be better option.
Even if we lack some necessary skills
It is not a secret that I lack some necessary skills too :) so it is not a great obstacle. Perhaps when/if the code becomes more public, there should be some docker image and instruction "how to launch it at your computer or anywhere else".
I've been playing around with PHP/CSS/mySQL on a free server
Actually this old version (mentioned above, at sourceforge) should be pretty fit for deploy at free server. This should be the same with current version. The site is so to say "lightweight", only features which require sending external requests will be broken (like login via 3rd party etc). So playing with it, fixing small issues etc should possible with no great skill.
To conclude however, no work was done by me yet to prepare code for opensourcing etc - but for the reason that I have recently idea which I'll try to describe shortly in different topic, in relation to the three small problems added by me recently.
I started grooming sources and here is the first "step" - they are stripped down to the bare central "framework" with
a single main page as a demo. Here are explanations of how it works, how to run it and suggestions of simple experiments
to try. One of the important thing we'd like to know is that the way pages are loaded with their "paths" specified in
the url, doesn't allow some hack like viewing (from user's point) any file's source, especially conf.php
or alike
which may contain settings, secrets etc.
https://github.com/codeabbey/src - anyone who feels curious enough to help with testing/review is quite welcome.
I already stuck upon few issues and made some amendments (one of the most annoying seems to be that site is going to be broken with future versions of the language due to "dynamic properties" on objects, but so it is good there is time to make fixes beforehand).
Next two steps I think would be adding basic pages and services which allow store, list and view problems - and then, users. In this form it will be already thing which everyone can deploy to start one's own problem-collection (e.g. for company, university or personal).
Hello, Rodion!
I want to help with testing/review, but the link you gave to us returns 404.
Is it possible to fix this?
With best regards, Qwerty.
Hi Qwerty :)
Yep, thanks and sorry - should be fixed now - either I'm complete idiot, or Github now switched to create private repos by default. Visibility should be changed to public now.
Also good news are that I pushed the next part of files, which provide user registration/login.
P.S. html-files (e.g. layout, pages, fragments), scripts and styles could be reused for templates in other languages with some modifications (I won't be surprised if there exist tools converting PHP template to Jinja2 template for Python or something like this). But probably you'll find it is simpler to deal with existing code without rewriting :)
Rodion, thank you! I quickly checked the code and docs and it looks very interesting.
However, my weekend is coming to end, so it is probably too late for me to make any progress.
Maybe someone else will try. If not, let's wait for next weekend.
No worry, no hurry :) my own progress is pretty slow - I only expect to add functionality for serving problems this week.
Meanwhile added small 3-rd part about .htaccess
files, which is quite auxiliary topic.
Small update - the 4-th part(or "step") was pushed to the sources now - it contains something you never seen on the web-site - functionality to add/edit tasks! Though here remains yet another step to allow users to solve these tasks (now only viewing them works).
As promised, though with small delay, here are two more updates - 5-th adds problem submitting/checking functionality and 6-th contains user profile page and ranking page.
Good news are that this completes "minimum functionality" of the site
E.g. potentially someone could grab the current sources as-is and deploy them to setup personal site for class, school, university or company. There are obviously some links leading to absent auxiliary functionality - they could be hidden (commented out etc).
I'll take some pause now to re-apply changes added in these updates back to the main site (as usually there were found some obsolete features and on contrary something was added) - and perhaps shall deploy it on free hosting and record "how to" video.
So, finally got some free time to deploy the web site.
To deploy web site, a VDS (virtual dedicated server) was chosen, as deploying web site is a megaton of complexities, so it would be nice to have complete control over the installed software.
I paid some money for the hosting and got credentials to access the server through SSH.
To work with web site through SSH, a program PuTTY was installed. I got PuTTY from here.
In the PuTTY window, we set up the connection.
Host Name (or IP address):
89.111.132.97
Port:
22
Connection type:
SSH
After clicking "OK", the following window were popped out:
The host key is not cached for this server:
89.111.132.97 (port 22)
...
After carefully checking the address and port, I clicked "Accept".
The invitation appeared: "login as" - entered root
Next question: "root@89.111.132.97 password" - entered utumh91ny6co
We see "Welcome to Ubuntu 22.04 LTS" - this is the operating system installed on the server.
Checking if the necessary minimum was installed:
root@vm2406295323:~# which nginx
root@vm2406295323:~# which apache
root@vm2406295323:~# which php
root@vm2406295323:~# which mysql
root@vm2406295323:~#
Okay, so we have none of the prerequisites installed. Let us try to install the required software.
For NGINX, I used the instructions from here (change ru
to en
in the link if appropriate).
Trying to install cURL:
root@vm2406295323:~# apt install curl
...
curl is already the newest version (7.81.0-1ubuntu1.16).
Installing gnupg2:
root@vm2406295323:~# apt install gnupg2
...
The following NEW packages will be installed:
gnupg2
...
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 gnupg2 all 2.2.27-3ubuntu2.1 [5548 B]
...
Trying to install ca-certificates:
root@vm2406295323:~# apt install ca-certificates
...
ca-certificates is already the newest version (20230311ubuntu0.22.04.1).
Trying to install lsb-release:
root@vm2406295323:~# apt install lsb-release
...
lsb-release is already the newest version (11.1.0ubuntu4).
lsb-release set to manually installed.
Trying to install ubuntu-keyring:
root@vm2406295323:~# apt install ubuntu-keyring
...
ubuntu-keyring is already the newest version (2021.03.26).
ubuntu-keyring set to manually installed.
Importing NGINX signing key:
root@vm2406295323:~# curl https://nginx.org/keys/nginx_signing.key \
| gpg --dearmor \
| tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null
% Total % Received ...
100 11809 100 11809 ...
Checking imported key:
root@vm2406295323:~# gpg --dry-run --quiet --no-keyring --import --import-options import-show /usr/share/keyrings/nginx-archive-keyring.gpg
pub rsa4096 2024-05-29 [SC]
8540A6F18833A80E9C1653A42FD21310B49F6B46
uid nginx signing key <signing-key-2@nginx.com>
pub rsa2048 2011-08-19 [SC] [expires: 2027-05-24]
573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62
uid nginx signing key <signing-key@nginx.com>
pub rsa4096 2024-05-29 [SC]
9E9BE90EACBCDE69FE9B204CBCDCD8A38D88A2B3
uid nginx signing key <signing-key-3@nginx.com>
The key 573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62 matches.
Setting up the repository:
root@vm2406295323:~# echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \
http://nginx.org/packages/ubuntu `lsb_release -cs` nginx" \
| tee /etc/apt/sources.list.d/nginx.list
deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] http://nginx.org/packages/ubuntu jammy nginx
Pinning the packages from repository:
root@vm2406295323:~# echo -e "Package: *\n \
Pin: origin nginx.org\n \
Pin: release o=nginx\n \
Pin-Priority: 900\n" \
| tee /etc/apt/preferences.d/99nginx
Package: *
Pin: origin nginx.org
Pin: release o=nginx
Pin-Priority: 900
Updating repository information:
root@vm2406295323:~# apt update
Get:1 http://nginx.org/packages/ubuntu jammy InRelease [3590 B]
Get:2 http://nginx.org/packages/ubuntu jammy/nginx amd64 Packages [21.6 kB]
...
Installing NGINX:
root@vm2406295323:~# apt install nginx
...
The following NEW packages will be installed:
nginx
...
Unpacking nginx (1.26.1-2~jammy) ...
Setting up nginx (1.26.1-2~jammy) ...
Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /lib/systemd/system/nginx.service.
...
Checking if NGINX installed:
root@vm2406295323:~# which nginx
/usr/sbin/nginx
Enabling NGINX service:
root@vm2406295323:~# systemctl enable --now nginx
Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable nginx
Checking NGINX service status:
root@vm2406295323:~# systemctl status nginx
● nginx.service - nginx - high performance web server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2024-07-13 12:57:49 MSK; 1min 6s ago
...
Process: 2304 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=0/SUCCESS)
Main PID: 2305 (nginx)
...
CGroup: /system.slice/nginx.service
├─2305 "nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf"
└─2306 "nginx: worker process" ...
At this point, we can visit http://89.111.132.97/ and see the NGINX welcome page.
Next step is installing PHP.
For PHP, I used these instructions. There may be good alternatives on English, or you can try Google translate this page linked above.
Installing prerequisites for PPA packages:
root@vm2406295323:~# apt install software-properties-common
...
The following additional packages will be installed:
python3-software-properties
The following packages will be upgraded:
python3-software-properties software-properties-common
...
Do you want to continue? [Y/n] y
...
Setting up python3-software-properties (0.99.22.9) ...
Setting up software-properties-common (0.99.22.9) ...
...
Installing PHP PPA package:
root@vm2406295323:~# add-apt-repository ppa:ondrej/php
...
Description:
Co-installable PHP versions: PHP 5.6, PHP 7.x, PHP 8.x and most requested extensions are included. Only Supported Ubuntu Releases (https://wiki.ubuntu.com/Releases) are provided.
...
CAVEATS:
1. If you are using php-gearman, you need to add ppa:ondrej/pkg-gearman
2. If you are using apache2, you are advised to add ppa:ondrej/apache2
3. If you are using nginx, you are advised to add ppa:ondrej/nginx-mainline or ppa:ondrej/nginx
...
Adding repository.
...
Installing NGINX PPA package (not sure we need it, just following the advice from previous output):
root@vm2406295323:~# add-apt-repository ppa:ondrej/nginx
...
Description:
This branch follows latest NGINX Stable packages compiled against latest OpenSSL for HTTP/2 and TLS 1.3 support.
...
Adding repository.
Updating repository information:
root@vm2406295323:~# apt-get update
...
Hit:6 https://ppa.launchpadcontent.net/ondrej/nginx/ubuntu jammy InRelease
Hit:7 https://ppa.launchpadcontent.net/ondrej/php/ubuntu jammy InRelease
Checking the latest version of PHP:
Installing PHP (substitute x.y
with values from previous step):
root@vm2406295323:~# apt-get install phpx.y
...
Creating config file /etc/php/8.3/mods-available/...
...
Setting up php8.3-cli (8.3.9-1+ubuntu22.04.1+deb.sury.org+1) ...
update-alternatives: using /usr/bin/php8.3 to provide /usr/bin/php (php) in auto mode
update-alternatives: using /usr/bin/phar8.3 to provide /usr/bin/phar (phar) in auto mode
update-alternatives: using /usr/bin/phar.phar8.3 to provide /usr/bin/phar.phar (phar.phar) in auto mode
...
Creating config file /etc/php/8.3/cli/php.ini with new version
...
Warning: Could not load Apache 2.4 maintainer script helper.
...
Installing PHP-FPM:
root@vm2406295323:~# apt-get install php8.3-fpm
...
The following NEW packages will be installed:
php8.3-fpm
...
Setting up php8.3-fpm (8.3.9-1+ubuntu22.04.1+deb.sury.org+1) ...
...
Creating config file /etc/php/8.3/fpm/php.ini with new version
Created symlink /etc/systemd/system/multi-user.target.wants/php8.3-fpm.service → /lib/systemd/system/php8.3-fpm.service.
...
Enabling PHP-FPM service:
root@vm2406295323:~# systemctl enable --now php8.3-fpm.service
Synchronizing state of php8.3-fpm.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable php8.3-fpm
Now we want to test PHP installation, and all instructions I found online instruct to place info.php file in /var/www for testing purposes:
root@vm2406295323:~# echo '<?php phpinfo();' > /var/www/info.php
-bash: /var/www/info.php: No such file or directory
Oops! The directory not exists! Did I misconfigured something?
I created the "www" folder:
root@vm2406295323:~# mkdir /var/www
root@vm2406295323:~# echo '<?php phpinfo();' > /var/www/info.php
root@vm2406295323:~#
Now I moved to http://89.111.132.97/info.php and there is a 404 error. Indeed, my configuration is wrong.
In order to fix that issue, I tried steps from here, starting from section "Configure Nginx with PHP".
root@vm2406295323:~# nano /etc/nginx/sites-available/default
[ Directory '/etc/nginx/sites-available' does not exist ]
Oops, all these nonexistent directories and files gets scary! I hope that I don't go wrong with making these...
root@vm2406295323:~# mkdir /etc/nginx/sites-available
root@vm2406295323:~# nano /etc/nginx/sites-available/default
I created the basic server configuration file:
http {
server {
}
}
And entered Ctrl + X
to exit the editor and save the file.
I also added /etc/nginx/sites-available
to the list of directories where NGINX should look for configuration files.
Opened the NGINX configuration file:
root@vm2406295323:~# nano /etc/nginx/nginx.conf
And added the following line to the end of http
context:
include /etc/nginx/sites-available/*;
Now I returned to default
server configuration file and continued to edit the configuration:
root@vm2406295323:~# nano /etc/nginx/sites-available/default
In the server
block, I added the following code:
listen 127.0.0.1:80;
root /var/www;
location / {
index index.php index.html index.htm;
}
location ~ \.php$ {
fastcgi_index index.php;
fastcgi_pass unix:/run/php/php8.3-fpm.sock;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
}
Then I restarted the server:
systemctl restart nginx
And checked http://89.111.132.97/info.php and there is still a 404 error.
I even tried to test a simple html file (no PHP at all!):
root@vm2406295323:~# nano /var/www/index.html
And entered:
<html><head><title>test</title></head><body><h1>test</h1></body></html>
And tested http://89.111.132.97 and it still not working (only standard NGINX welcome page is displayed).
I opened the error log:
root@vm2406295323:~# cat /var/log/nginx/error.log
And there are the following lines:
2024/07/13 16:03:12 [error] 13749#13749: *1 open() "/usr/share/nginx/html/info.php" failed (2: No such file or directory), client: 46.151.156.96, server: localhost, request: "GET /info.php HTTP/1.1", host: "89.111.132.97"
Wait! Be we set up /var/www as root of our web site, why it is looking in /usr/share/nginx/html???
Let us grep through the server to find there /usr/share/nginx/html is set:
root@vm2406295323:~# grep -inr '/usr/share/nginx/html' /
/etc/nginx/conf.d/default.conf:8: root /usr/share/nginx/html;
/etc/nginx/conf.d/default.conf:18: root /usr/share/nginx/html;
Okay, let us look through that file as well:
root@vm2406295323:~# nano /etc/nginx/conf.d/default.conf
Yeah, this file contains the line:
root /usr/share/nginx/html;
And this explains why it looks in /usr/share/nginx/html
for files, though it not explains why configuration in /etc/nginx/sites-available/default
is ignored.
I deleted this line and replaced the contents of the /etc/nginx/conf.d/default.conf
file with the contents of /etc/nginx/sites-available/default
file.
Then I deleted the /etc/nginx/sites-available
folder because it was ignored anyway:
root@vm2406295323:~# rm -rf /etc/nginx/sites-available
I also removed this line from main NGINX configuration file, since it was ignored anyway:
include /etc/nginx/sites-available/*;
Then I restarted the server, and this time it failed to restart successfully:
root@vm2406295323:~# systemctl restart nginx
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.
And the status is:
root@vm2406295323:~# systemctl status nginx.service
× nginx.service - nginx - high performance web server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2024-07-13 16:46:38 MSK; 39s ago
....
Jul 13 16:46:35 vm2406295323 systemd[1]: Starting nginx - high performance web server...
Jul 13 16:46:35 vm2406295323 nginx[14024]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
According to https://stackoverflow.com/questions/14972792/nginx-nginx-emerg-bind-to-80-failed-98-address-already-in-use
I tried the line:
root@vm2406295323:~# pkill -f nginx & wait $!
[1] 14139
[1]+ Done pkill -f nginx
root@vm2406295323:~# systemctl start nginx
Okay, no errors this time, so I tried to visit http://89.111.132.97. And imagine what? It still shows me the NGINX default page!
I googled and googled and googled and finally found the trick: press Ctrl + F5
in browser and it will show you the actual content of the page, not the cached version.
However, http://89.111.132.97/info.php now returns "502 Bad Gateway". Still not working.
Let us see what we have in error.log:
2024/07/13 17:00:59 [crit] 14150#14150: *4 connect() to unix:/run/php/php8.3-fpm.sock failed (13: Permission denied) while connecting to upstream, client: 46.151.156.96, server: localhost, request: "GET /info.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.3-fpm.sock:", host: "89.111.132.97"
Well, maybe instead of unix:/run/php/php8.3-fpm.sock we must try some other line. Let us try this line instead:
fastcgi_pass localhost:9000;
It is not working too, accorging to error logs:
2024/07/13 17:15:45 [error] 14633#14633: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 46.151.156.96, server: localhost, request: "GET /info.php HTTP/1.1", upstream: "fastcgi://[::1]:9000", host: "89.111.132.97"
Well, let us try to work on /run/php/php8.3-fpm.sock solution again. This is one of the discussions I found on this topic:
https://stackoverflow.com/questions/23443398/nginx-error-connect-to-php5-fpm-sock-failed-13-permission-denied
According to that discussion, I added nginx to the www-data group:
root@vm2406295323:~# usermod -a -G www-data nginx
And returned the previous setting:
root@vm2406295323:~# nano /etc/nginx/conf.d/default.conf
Namely, the line:
fastcgi_pass unix:/run/php/php8.3-fpm.sock;
And restarted NGINX and PHP-FPM:
root@vm2406295323:~# systemctl restart nginx
root@vm2406295323:~# systemctl restart php8.3-fpm.service
Wow! The page http://89.111.132.97/info.php is working now!
Sorry I still not tested anything of CodeAbbey's site today. It was quite a work to do even a basic configuration of web site.
Hi Qwerty!
If you don't mind, it would be better perhaps to create "issue" directly in the repository and put your notes there - as I doubt this thread is well suited for technical details.
Honestly I feel you are doing it the hardest possible way :) I don't know much about nginx
(though it is used for sandbox
for example) - but few minor things of this solution depend on Apache httpd
as a web-server (I get used to think that
nginx
is more often met for proxy etc).
You could reuse the dockerfile in the repo for deploying the stuff on VPS (though of course it makes sense to extract database out of the container).
Also I'm unsure about the idea of using VPS/VDS for the website of such kind - as you'll need to solve a number of technical issues (you see, starting from installing the stuff - and ending with https certificates, ddos-protection etc), so I myself keep it on old-style shared hosting not without a reason. While again sandbox uses VPS.
That said, I learnt that our colleague Kevin (moxieman) had success with setting this up at awardspace (free shared hosting which we use for interactive problems by the way).
Hi all!
Yes, I was able to find some success in deploying on Awardspace. It was relatively straightforward (though maybe a bit tedious) to upload the various files and set up the directories, and they also offer integrated mySQL for free accounts. Setting up the SQL database is only slightly more complicated, but isn't too bad once you get a feel for how the interface works. Overall it seemed a bit easier than the process qwerty
is suffering through above...
Free accounts do expire after a year of inactivity though, so just be aware of that!
Hello everyone,
I finally installed and configured all the required software.
Then started with https://github.com/CodeAbbey/src/blob/v0.1-framework/index.php
I was really excited about the possibilty to learn from an experienced programmer's code, but after several hours of studying code in index.php
realised that this is much above my level...
So, unlike Kevin aka moxieman
, I was unable to deploy even a single file (index.php) to the site (Kevin, you are a real hero and the best progrmmer!).
Here are the moments I could not understand:
In the prepare
function, we see:
addFile('conf.php');
What will happen if conf.php is missing? I do not see any special processing of false
return value.
new \module\sys\Elems();
...do not bind the result to any value...
Why we create a class instance and then discard it? What is the purpose of the new \module\sys\Elems();
line?
$ctx = new \module\Context();
...
$ctx->elems->path = preg_replace('/^(.*\/)[^\/]*$/', '$1', $_SERVER['PHP_SELF']);
..but the class \module\Context does not contain the elems
property.
Shouldn't that lead to error?
I made a test code:
<?php
$ctx = new \stdClass();
$ctx->elems->path = '';
echo $ctx->elems->path;
And after running it, I got:
Warning: Creating default object from empty value in the code at line 3
I tried to look into it a bit more and the class extends \module\sys\ProtoContext which have:
function __get($name) {
$v = $this->names[$name] ?? null;
if ($v !== null) return $v;
$methodName = 'get' . ucfirst($name);
if (!method_exists($this, $methodName)) {
throw new \Exception("No property '$name' in Context!");
}
$res = $this->$methodName();
if (is_object($res)) {
$res->ctx = $this;
}
$this->names[$name] = $res;
return $res;
}
I guess that we call:
protected function getElems() {
return \module\sys\Elems::$elems;
}
When we are trying to set $ctx->elems->path
- right?
Even if I am right, very complex way of doing things...
What is the meaning of this?
preg_replace('/^(.*\/)[^\/]*$/', '$1', $_SERVER['PHP_SELF']);
I am not good at regular expressions, can you add a PHP comment here to describe the purpose?
$page = isset($_GET['page']) ? $_GET['page'] : 'main';
$page = preg_replace('/[^a-z0-9\_]/', '', $page);
$page = str_replace('_', '/', $page);
I tested the original CodeAbbey site and did not found where 'page' is used in the request. Can you give an example?
Wouldn't there be any problems that addfile
function is declared with all lowercase letters but called with addFile
and addfile
?
Or PHP is a case insesitive language (sorry for maybe silly question, I am not famillar with the language)?
In the addfile
function...
function addfile($name) {
global $model, $ctx;
if (!file_exists($name)) {
return false;
}
include($name);
return true;
}
...I cannot understand the global $model, $ctx;
here, as there are no usages.
But I already understood that PHP is very complex language, so maybe some subtle side effects of these declarations?
$r = addfile($ctlFile);
Can anyone explain the assignment above? I do not see any usage of $r
...
addFile("ctl/_hookPreRender.php");
I did not found _hookPreRender.php.
array_push($ctx->elems->styles, $ctx->elems->page);
array_push($ctx->elems->scripts, $ctx->elems->page);
I do not get how a page can serve itself both as stylesheet and as a script? Can anyone elaborate a bit?
What is this?
ob_start();
...
$ctx->elems->contentResult = ob_get_clean();
I tried to read:
https://www.php.net/manual/ru/function.ob-start.php
But did not understand much...
addFile("ctl/_hookAfter.php");
I did not found _hookAfter.php.
What is the func_num_args
in the url
function?
Does that mean that url
function can accept some arguments not described in the function definition?
What is these arguments and why not explicitly add these arguments to the definition to make programmer's life easier?
} else if ($numArgs > 3) {
$i = 1;
$args = func_get_args();
$query = array();
while ($i < $numArgs - 1) {
if ($args[$i] !== null && $args[$i + 1] !== null) {
array_push($query, $args[$i] . "=" . $args[$i + 1]);
}
$i += 2;
}
$res .= ($rewrite ? '?' : '&') . implode('&', $query);
}
This is not easier than the code above? https://www.php.net/manual/en/function.http-build-query.php
function aurl($url)
It is not clear from the name what is difference between url
and aurl
functions, maybe choose another name to help me understand...
spl_autoload_register(function($class) {
$path = str_replace('\\', '/', $class) . '.php';
if ((@include $path) === false) {
echo "Loading $class file failed!\n";
} elseif (!class_exists($class, false) && !interface_exists($class, false)) {
echo "Loading $class failed!\n";
}
});
I do not understand to where errors are echoed. Directly to the page shown to user?
Sorry it seems that I won't get the site running for a while, maybe for 3-4 months.
I am just an unexperienced and very low-skilled programmer, it took me a good five hours to read the index.php file, so you can imagine how much time it will take for me to read all files in the reposiory...
Thankfully, Kevin seems to be much more experienced programmer, so he somehow handled all these issues - amazing work!
Qwerty, Hi! first of all, don't regard it as the "code by experienced user". I don't have PHP experience besides this site and it was mostly written 10 years ago when I had much less experience in anything at all.
Hopefully we'll improve and clarify it by and by, but generally it is not great example for study.
Again I recommend to ask questions about code using GH issues, it's more handy than here.
Deploy all the files of the step 1 and see if you can make it work. Much understanding of the code mainly is needed if you nead to tweak it.
aulr is for abs.url, rarely used. Elems set static field in constructor (a bit confusing, yes). addFile allows file to be missing, so it is ok that some files are not there yet. error handling is very poor. you may in some cases need to look for them in page source (ctrl-u in browser).
I'll return with other answers, hopefully in gh issues, but note that normally one is not supposed to have deep knowledge of the language to use this (otherwise it will be useless). I think Kevin has just basic knowledge of PHP and I myself won't recollect some things without googling :)
Hi qwerty
! I would agree with Rodion
- I'm hardly an experienced PHP
programmer! I wrote my first line of PHP less than five months ago to solve the Dynamic Web Page problem. I've had a little PHP
practice in writing checkers for problem submissions and also playing around with SQL interfacing, but these files are the first time I'm seeing a lot of "real" web dev concepts (login system, js integration, global PHP
objects, etc).
Luckily as Rodion
mentioned, "knowing how it works" isn't really required for "getting it to run". I'd recommend just uploading the files and getting it working "like magic" first, then start playing around with lines of code to see what the effects are. Thankfully everything is already set up to work "as-is" if you just download all the files directly from the GitHub repository then upload to the hosting server (except for very simple tweaks of conf.php
in the later branches to define the SQL login credentials).
I certainly couldn't explain very well what every line of index.php
is doing, but I'm slowly working my way through it and trying to figure it out as I go. So don't worry if much of the code details seem quite confusing at this point! I empathise with wanting to understand how everything is working from the start, but I've never met anyone who learned how to work on car engines without first having much experience driving :)
I can try to answer some of your PHP
-specific questions...
PHP
is a "partially" case-sensitive language. Variable names are case-sensitive, but function names are not. So $myvar
and $MyVar
are unique, but myfxn()
and MyFxn()
are the same.
Regarding those 4 regexp lines...
page
and defaulting to main
otherwise, andHowever note that looking into this raises further questions for me (santizing url .../iNdEx/TaSk_LiSt
to .../index/task_list
works for my mirror site but apparently not the main site? wouldn't the underscore in task_list
get converted to a forward slash? is that even what those lines are really doing?) but the main takeaway here should be that those questions don't need to be answered to run the site, and now that I have a "partially-magic" running site to play with I could probably answer those questions by taking some time, commenting out lines of code, and generally playing with things to see their effects :)
Variable names are case-sensitive, but function names are not
Funny. I never know until today :)
BTW, Kevin, if you would like, I also recommend to put questions into issues in GH - just right now it looks we are collecting more and more of them and answering some of them requires some "archeology" work in the code - despite the "core" or framework is pretty small (hopefully) I definitely don't remember how some of these things evolved :)
About ob_start
. See, normally PHP page works like HTML with insertions of small scripting elements (in php
tags).
Of course it long since changed and particularly here we use the full-blown script or application in PHP
code which needs at some point to load other PHP which mainly contains HTML (e.g. page templates), decorate it with
some "layout", insert some "fragments" and only then send to output. Regretfully when we include some file, it is
sent to output immediately, so we use this queer magic - ob_start
instructs interpreter to stop sending things into
output, but other put them to some "output buffer" (I guess hense the prefix "ob"). When the template is thus "collected"
we get it from the buffer and process at our will. This is definitely confusing because some errors in the code before
this may be also sent to output, breaking the page, headers and everything.
Kevin, Rodion, first of all I want to thank you for great answers! Learned a lot from these.
Second, I understand that a GitHub issue is a better place to continue discussion, but I am not an experinced GitHub user yet (my only experinece is that python-django codeabbey repository linked above). Perhaps I will try to make my first issue with more free time in disposal.
For now, the following was done:
Cloned the latest version repository to my local computer, so I can make any needed changes in the code before uploading it to site.
git clone https://github.com/CodeAbbey/src.git
Not making any changes in the code yet, installed WinSCP and transfered all the code to /var/www folder of the server.
Of course, this did not work right away: when I tried to load http://89.111.132.97/index.php, the message appeared: "Loading mysqli file failed!"
I checked README.md, navigated to http://89.111.132.97/sqlexec.php and inserted the contents of dbinit.sql into text field (I did not make any changes to the sql)
I got: Result: fail (127) sh: 1: mariadb: not found.
I ran apt update
then apt install mariadb-server
.
Tried to execute SQL again and got: Result: fail (1) ERROR 1698 (28000): Access denied for user: 'www-data'@'localhost'
This time I do not fully understand the error, can anyone give me hints on how should I debug this error?
By the way, www-data
is the user who runs PHP code on the server.
I checked the code in the sqlexec.php and understood that it runs a quite simple command.
So I just copied the dbinit.php file to the home folder of root user and run:
cat dbinit.php | mariadb -B
This worked like a charm, the database is created as well as all tables.
And I fixed the error: "Loading mysqli file failed!" by running the command:
apt-get install php8.3-mysqli
However, now I'm getting a HTTP 500 error at http://89.111.132.97/index.php.
The NGINX error log says:
Uncaught mysqli_sql_exception: You have an error in your SQL syntax: ... near the ')' at line 1 in /var/www/module/dao/MysqlDao.php:67
Qwerty, Hi!
As about github issues - just register on GH, navigate to the project page, click "issues" and then green button "New Issue". the same markdown format is used, but with some additional things like cross-referencing issues, linking code etc. If the site is down it will be difficult to you to find all these notes if they are hosted on the site itself :)
Installing mariadb and workaround for the user to run mysql with sqlexec could be found in Dockerfile in the root folder. Even though you decided not to use docker, there are all the commands I myself used to install things so it may be helpful :)
Error itself looks queer, as you understand without some more debugging efforts on your side it is hard to tell what
may be wrong. Open this file at this specific line, wrap it with try ... catch
(you'll easily google for syntax - I
don't remember myself) - and log the query if there is error perhaps. Or you can enable query logs in mysql itself.
BTW the "latest version" may contain some intermediate changes... "step-X" branches should be more stable, I think.
Have you configured anything about DB in conf.php
? Or defaults worked for you?
Ah, also you can probably change error reporting with the first line in conf.php
(one with "display_errors") - though
still you may need to look into page source after loading to find them.
Hi, the issue was due to recent changes reacting to empty task list (so really master branch should be used with caution)
Either update files (there are fixes in master now) or populate db with queries from the taskinit file.
particular error was due to SQL clause ...id in ($task_id_list)...
which unexpectedly is "incorrect syntax" in mysql when parentheses are empty.
Hello, Rodion,
Just uploaded the latest version of sources to server, and everything works! Hurrah!
By the way, I tried to debug the issue yesterday, but with no avail.
Here is how I tried to debug:
function find($cond = null, $limit = null, $offset = null) {
// Added the line below after reading
// https://stackoverflow.com/questions/27345377/try-catch-in-mysqli
mysqli_report(MYSQLI_REPORT_STRICT | MYSQLI_REPORT_ALL);
$query = "select * from {$this->table}";
$query .= $this->clauses($cond, $limit, $offset);
try {
$res = $this->conn->query($query);
}
catch (mysqli_sql_exception $e) {
fwrite(
STDERR,
"In /var/www/module/dao/MysqlDao.php:find. Query is: $query\n"
);
file_put_contents(
"/var/www/crash.log",
"In /var/www/module/dao/MysqlDao.php:find. Query is: $query\n"
);
throw $e;
}
return $this->objectsArray($res);
// Other topics that did not helped me (but may give you an idea):
// https://www.php.net/manual/ru/mysqli-driver.report-mode.php
// https://stackoverflow.com/questions/72241551/why-mysqli-sql-exception-is-not-caught-by-php
}
But PHP just kept ignoring all error handling: no messages to stderr, nothing written in crash.log.
After several hours trying to properly set up the error handling, I finally gave up. In Python it was much easier to handle errors...
If anyone knows what was wrong with the above code, I would be grateful to know.
Hi Qwerty!
I'm not that deep in PHP so it's the first time I hear of specific mysqli_sql_exception
:) My code is simpler:
try {
$res = $this->conn->query($query);
} catch (\Exception $e) {
echo "Exception on query $query";
}
it may be good to set $res = [];
probably after error is caught to prevent further errors. anyway it works for me
and you can test it by temporarily breaking the query in ($taskIds)
in the ctl/Main.php
for example (e.g. remove
variable leaving empty parentheses).
I'm not sure if there could be some server-side options for PHP preventing exception handling (for errors they exist, see above).
Somewhat important update - step 8 - here we start taking care of customization needs - e.g. if someone of you deploy your own version of the site, you want to change its name, front page elements etc - for this we'll be providing dedicated settings etc (so that many important changes could be done in a single place and without messing in the sources).
Added steps 9 and 10 (task tags and wiki pages)...
and more important:
Recorded Video Instruction of deploy to AwardSpace - so that now anyone should be able to clone and setup the site in just half an hour - this doesn't require special knowledge, but there are few subtle points (I stumbled upon them myself).
Hello, Rodion,
Good thing, it is a weekend again!
What should I do today, after deployment the site? As far as I can remember, you opensourced CodeAbbey in order for us to help finding vulnerabilities, right?
What exact vulnerability I should seek? I am by no way a security specialist, so please forgive me my silly question.