Communization of CodeAbbey

Back to General discussions forum

Rodion (admin)     2024-05-31 12:03:33
User avatar

Perhaps "communization" is not the best term or may have unnecessary negative connotations - feel free to propose topic title improvement.

While our site is not in top-10 of the whole internet, it still have some audience and some content, in creation of which many people have participated. Some years ago I started thinking how best to preserve it in case of anything unexpected. Recently it was mentioned that the very least effort should be publishing the problem statements separately so they won't suddenly perish. Our colleague Clive contacted me expressing the thought that something more than "least effort" could be anticipated, like sharing or delegating administrative functions etc. I thought of such possibility often but failed to come to some good idea (mainly about how to do it logically/technically and who may be interested in being involved).

Now let me outline main points as I see them with the goal that you can review and perhaps suggest some additions or improvements.

  1. What goals we may have? Firstly - that in the event of something happening to me or to the site, some initiative colleagues have access to materials needed to redeploy it somewhere else. Secondly - perhaps that some administrative duties, like accepting new tasks - could be done by some elect members.

  2. As website consists mainly of two entities - source code and data in the database - how can they be shared (in order the site could be recreated without myself)? For the code most obvious answer seems to be "opensource it", while data has some things which should be kept relatively secretly (user authentication data, problem checking scripts) - so probably data backups should be made periodically and access to them should be given to few trusted persons. There are other auxiliary things (sandbox, server and code for interactive problems etc) - but let's ignore this for now just not to add complexity.

  3. Opensourcing the site code is technically easy - just copying it from the private repository - main hampering factor is that there supposedly could be some security flaws and making them public brings some more risks. However there seem to be not much which could be stolen from the site in case of some breach exploited. I can't see remedy for this except careful and somewhat tedious review process. Thus probably it is going to happen in steps: firstly code is copied bit by bit into another private repository to which more people (but still limited number) have access with the goal of trying it, reviewing it and recommending any necessary improvements and removals. When the whole thing is less or more reviewed, that repo is open to public. Technically from this point any enterprise or university can fetch the code and deploy its own clone for own needs.

  4. Management of tasks sounds perhaps more tricky - I can add some logic that chosen people can be given some "guru" rights - they can add new tasks, invisible, test the setter/checker code, refine statement and then make it visible. The main drawback is that currently here is zero "collaborative" features, so such "gurus" need to be careful not to override each one additions etc. Also it is a question whether actions of one guru need to be approved by other(s). Or on contrary whether one can prevent/cancel other's actions if anything went wrong.

Please feel free to discuss or express your wishes if something important is forgotten!

qwerty     2024-06-04 11:12:09
  1. I may redeploy the site on a hosting of your choice or on the hosting I currently use. Administrative duties like accepting new tasks - could be done by members with certificate, I think, because they have the certificate anyway, and are not interested in adding exercises just for their favor.
  2. I agree, it does not seem good to disclose database contents. You must backup database on regular basis, but do not disclose it to anyone (even to trusted persons, I think).
  3. I can create a public repository and slowly build up a clone of site on that repository using language of your choice and database engine of your choice. This way we will have the opensourced code and you will not need to disclose any of your source code. But this work will take some time, so please be patient. I also expect others to review the work and suggest any necessary improvements, otherwise I may refuse to continue.
  4. Management of tasks should be done through issues, I think. When there is a task to be done, we create an issue on public repository, and if someone is ready to work on this issue, he/she will be appointed to work on that issue. There will be only one working on a given issue at a time, so no risk of overwrite.
qwerty     2024-06-10 07:03:02

Some work was done this weekend:

  1. Source code now available here
  2. Documentation is available here
  3. Demo is available here

However, I lack frontend development skills, so any help on building a responsive design is greatly appreciated.

Rodion (admin)     2024-06-10 21:24:45
User avatar

Hi Friend!

This is impressive, but I pray not to go too far without synchronising our intentions :)

I'm sorry for I failed to response in time - I thought some more people may join discussion and meanwhile tried some different direction (I'll share about it bit later).

Reviewing your points:

  1. I'd prefer not to investing our time on rewriting the site to different language. It is of course possible, but not very practical. Though site is not a facebook of course, it is significant amount of functionality and thus it would be quite a bulk of work. The second reason is that problem setting/checking code for most of tasks is executed by the same engine and hence this code is written in PHP. Of course it would be possible to call PHP from Python but considering also that PHP hosting is generally easier found - this leaves not too much reason to dive into the work of recreating the site in Python or anything else (though I agree if choose among other languages, Python is preferable over others as it is the most popular language among the users of the site).
  2. Sharing DB backups with someone is however necessity due to obvious reason - if something happens to me, someone should have access to the data to recreate the site anywhere.
  3. There was OpenAbbey initaitive and its code could be found in github, but it didn't get much traction. The same seemingly is situation with idea of opensourcing current site code (old version, as I mentioned, is here and may be useful for assessing/reference). So perhaps I need yet to think of some smarter approach.
  4. Management of tasks just by anyone having certificate sounds a bit dangerous - as this means not only adding new problems but also editing existing. That's why I thought it should be limited to small number of people (unless we somehow switch to idea of creating tasks by virtually everyone, which is also possible but requires some different way of listing/searching through them).

I lack frontend development skills

Me too, though of course existing scripts/styles and templates could be shared. Regretfully templates are also in PHP and if you stick to idea of doing this in Python you'll need to change them to jinja or something alike, as I understand.

Concluding During the past week or two watching the lack of excitement about this activity :) I gradually have come to slightly different / alternative approach which I hope to uncover in a few days. Please get tuned, there'll be something to lay our hands on and wrap our minds about!

Quandray     2024-06-11 05:57:10
User avatar

I'll help, if there's things I can do

gardengnome     2024-06-11 07:13:00
User avatar

Good morning Rodion,

I don’t think there is a lack of willingness to help, rather I feel we are uncertain how exactly we can help.
I am most certainly keen to contribute in some way, but my total lack of web development experience somewhat limits direct practical support. But there are different aspects:

  • Support of day-to-day running of the site.
  • Support of backup process.
  • Financial support - I know that is not always straightforward in these times.
  • Maybe also sharing of all login details with one or two trusted parties as a further safety net.
qwerty     2024-06-11 11:08:37

This is impressive, but I pray not to go too far without synchronising our intentions :)

All right! I will wait until we have a clear understanding of our intentions.

I'd prefer not to investing our time on rewriting the site to different language.

I do not have the source code of original site, so I wasn't rewriting anything, just writing something from scratch. Of course, I understand the amount of work you have done, so if instead of reinventing the wheel I can help you with improving already written code, this would be much better.

By the way, thank you for mentioning OpenAbbey and linking the old site version. I will look into these, as it sound very interesting.

Management of tasks just by anyone having certificate sounds a bit dangerous.

No problem! We can then stick to current system, where programming problems are proposed through forum and reviewed by you or people you choose.

moxieman     2024-06-11 13:47:46
User avatar

I want to echo everybody that I am willing to help, but personally I'm somewhat unsure of my capacity to do so. I've been playing around with PHP/CSS/mySQL on a free server using what I've learned here as a launchboard, but this website seems a good deal larger in scale!

Either way, I'm willing to help in whatever way I might be able :)

qwerty     2024-06-11 17:53:30

You see how many responses, Rodion. Even if we lack some necessary skills, we really can do something together.

qwerty     2024-06-16 13:13:25

Sorry people, haven't much success in CodeAbbey development this weekend...

But studying abbey-2 and MessCode (SourceForge variant of CodeAbbey) is a really educational way to spend one's time :)

Rodion (admin)     2024-06-17 14:46:34
User avatar

Hi Friends, sorry for the delayed answer - yes now we see there are quite a "Mighty Handful" of colleagues - but as Mathias correctly pointed out it is bit difficult to understand what kind(s) of help are needed.

Not that I myself have quite clear idea yet. Let's look at the points raised by Mathias:

Support of day-to-day running of the site.

What "day-to-day" activities are necessary? Well, besides answering at forum (which already is done by you all very well) there are mainly two activities - adding new problems and handling too blatant cheaters. The second can be skipped for now - while how exactly to deal with the first - the simplest way is to allow certain users the access to add / edit problem interface. And create necessary improvements and instructions to it. Main concern is that has no features to simplify cooperative work. But we can invent them by and by.

Support of backup process.

Yes, probably it should be like the backups are created weekly, and for example, Graeme and Clive receive emails notifying backup is successful - and where to find it (some link probably). If the site is down for prolonged time and seemingly I'm unresponsive, backups are handed by them to some other members who will be willing to redeploy everything somewhere else. Though it is an open question how to share the domain name.

Financial support - I know that is not always straightforward in these times.

True but mostly it's ok for now - the most annoying thing is generally that different things are managed in different places. For the main server (Hostgator in US) I was able to make bank transfer for 3 years ahead, for sandbox (which is also used for checker in some "special" problems) they (BlueVPS in Europe) luckily accept cryptopayments, while domain name itself currently is in hands of BigRock (which is Indian). Simply explaining some of my foreign friends "login here, pay there" with clumsy interfaces of all these facilities makes more headache than the issue of spending money themselves :) It would be great probably to unite managing of all those first under some single service. We'll see.

Maybe also sharing of all login details with one or two trusted parties as a further safety net.

That is possible, just regretfully may be futile (if we speak of credentials to hosting, rather than administrative credentials of the site itself) - nowadays most logins from unusual location may require confirmation by email and logging in to email may require confirmation by phone. Additionally current hosting is used for some other of my small projects so it is somewhat messy. So probably mirroring and gradually shifting it to some carefully chosen hosting which allows properly sharing access could be better option.

Even if we lack some necessary skills

It is not a secret that I lack some necessary skills too :) so it is not a great obstacle. Perhaps when/if the code becomes more public, there should be some docker image and instruction "how to launch it at your computer or anywhere else".

I've been playing around with PHP/CSS/mySQL on a free server 

Actually this old version (mentioned above, at sourceforge) should be pretty fit for deploy at free server. This should be the same with current version. The site is so to say "lightweight", only features which require sending external requests will be broken (like login via 3rd party etc). So playing with it, fixing small issues etc should possible with no great skill.

To conclude however, no work was done by me yet to prepare code for opensourcing etc - but for the reason that I have recently idea which I'll try to describe shortly in different topic, in relation to the three small problems added by me recently.

Rodion (admin)     2024-06-28 15:55:40
User avatar

I started grooming sources and here is the first "step" - they are stripped down to the bare central "framework" with a single main page as a demo. Here are explanations of how it works, how to run it and suggestions of simple experiments to try. One of the important thing we'd like to know is that the way pages are loaded with their "paths" specified in the url, doesn't allow some hack like viewing (from user's point) any file's source, especially conf.php or alike which may contain settings, secrets etc.

https://github.com/codeabbey/src - anyone who feels curious enough to help with testing/review is quite welcome.

I already stuck upon few issues and made some amendments (one of the most annoying seems to be that site is going to be broken with future versions of the language due to "dynamic properties" on objects, but so it is good there is time to make fixes beforehand).

Next two steps I think would be adding basic pages and services which allow store, list and view problems - and then, users. In this form it will be already thing which everyone can deploy to start one's own problem-collection (e.g. for company, university or personal).

qwerty     2024-06-29 10:42:19

Hello, Rodion!

I want to help with testing/review, but the link you gave to us returns 404.

Is it possible to fix this?

With best regards, Qwerty.

Rodion (admin)     2024-06-30 06:20:11
User avatar

Hi Qwerty :)

Yep, thanks and sorry - should be fixed now - either I'm complete idiot, or Github now switched to create private repos by default. Visibility should be changed to public now.

Also good news are that I pushed the next part of files, which provide user registration/login.

P.S. html-files (e.g. layout, pages, fragments), scripts and styles could be reused for templates in other languages with some modifications (I won't be surprised if there exist tools converting PHP template to Jinja2 template for Python or something like this). But probably you'll find it is simpler to deal with existing code without rewriting :)

qwerty     2024-06-30 14:46:59

Rodion, thank you! I quickly checked the code and docs and it looks very interesting.

However, my weekend is coming to end, so it is probably too late for me to make any progress.

Maybe someone else will try. If not, let's wait for next weekend.

Rodion (admin)     2024-07-01 14:16:51
User avatar

No worry, no hurry :) my own progress is pretty slow - I only expect to add functionality for serving problems this week.

Meanwhile added small 3-rd part about .htaccess files, which is quite auxiliary topic.

Rodion (admin)     2024-07-03 23:10:52
User avatar

Small update - the 4-th part(or "step") was pushed to the sources now - it contains something you never seen on the web-site - functionality to add/edit tasks! Though here remains yet another step to allow users to solve these tasks (now only viewing them works).

Rodion (admin)     2024-07-08 19:27:51
User avatar

As promised, though with small delay, here are two more updates - 5-th adds problem submitting/checking functionality and 6-th contains user profile page and ranking page.

Good news are that this completes "minimum functionality" of the site

E.g. potentially someone could grab the current sources as-is and deploy them to setup personal site for class, school, university or company. There are obviously some links leading to absent auxiliary functionality - they could be hidden (commented out etc).

I'll take some pause now to re-apply changes added in these updates back to the main site (as usually there were found some obsolete features and on contrary something was added) - and perhaps shall deploy it on free hosting and record "how to" video.

qwerty     2024-07-13 15:10:32

So, finally got some free time to deploy the web site.

To deploy web site, a VDS (virtual dedicated server) was chosen, as deploying web site is a megaton of complexities, so it would be nice to have complete control over the installed software.

I paid some money for the hosting and got credentials to access the server through SSH.

To work with web site through SSH, a program PuTTY was installed. I got PuTTY from here.

In the PuTTY window, we set up the connection.

Host Name (or IP address):

89.111.132.97

Port:

22

Connection type:

SSH

After clicking "OK", the following window were popped out:

The host key is not cached for this server:
89.111.132.97 (port 22)
...

After carefully checking the address and port, I clicked "Accept".

The invitation appeared: "login as" - entered root

Next question: "root@89.111.132.97 password" - entered utumh91ny6co

We see "Welcome to Ubuntu 22.04 LTS" - this is the operating system installed on the server.

Checking if the necessary minimum was installed:

root@vm2406295323:~# which nginx
root@vm2406295323:~# which apache
root@vm2406295323:~# which php
root@vm2406295323:~# which mysql
root@vm2406295323:~#

Okay, so we have none of the prerequisites installed. Let us try to install the required software.

qwerty     2024-07-13 15:11:14

For NGINX, I used the instructions from here (change ru to en in the link if appropriate).

Trying to install cURL:

root@vm2406295323:~# apt install curl
...
curl is already the newest version (7.81.0-1ubuntu1.16).

Installing gnupg2:

root@vm2406295323:~# apt install gnupg2
...
The following NEW packages will be installed:
  gnupg2
...
Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 gnupg2 all 2.2.27-3ubuntu2.1 [5548 B]
...

Trying to install ca-certificates:

root@vm2406295323:~# apt install ca-certificates
...
ca-certificates is already the newest version (20230311ubuntu0.22.04.1).

Trying to install lsb-release:

root@vm2406295323:~# apt install lsb-release
...
lsb-release is already the newest version (11.1.0ubuntu4).
lsb-release set to manually installed.

Trying to install ubuntu-keyring:

root@vm2406295323:~# apt install ubuntu-keyring
...
ubuntu-keyring is already the newest version (2021.03.26).
ubuntu-keyring set to manually installed.
qwerty     2024-07-13 15:12:01

Importing NGINX signing key:

root@vm2406295323:~# curl https://nginx.org/keys/nginx_signing.key \
    | gpg --dearmor \
    | tee /usr/share/keyrings/nginx-archive-keyring.gpg >/dev/null
  % Total    % Received ...
100 11809  100 11809 ...

Checking imported key:

root@vm2406295323:~# gpg --dry-run --quiet --no-keyring --import --import-options import-show /usr/share/keyrings/nginx-archive-keyring.gpg
pub   rsa4096 2024-05-29 [SC]
      8540A6F18833A80E9C1653A42FD21310B49F6B46
uid                      nginx signing key <signing-key-2@nginx.com>

pub   rsa2048 2011-08-19 [SC] [expires: 2027-05-24]
      573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62
uid                      nginx signing key <signing-key@nginx.com>

pub   rsa4096 2024-05-29 [SC]
      9E9BE90EACBCDE69FE9B204CBCDCD8A38D88A2B3
uid                      nginx signing key <signing-key-3@nginx.com>

The key 573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62 matches.

Setting up the repository:

root@vm2406295323:~# echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] \
http://nginx.org/packages/ubuntu `lsb_release -cs` nginx" \
    | tee /etc/apt/sources.list.d/nginx.list
deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] http://nginx.org/packages/ubuntu jammy nginx

Pinning the packages from repository:

root@vm2406295323:~# echo -e "Package: *\n \
Pin: origin nginx.org\n \
Pin: release o=nginx\n \
Pin-Priority: 900\n" \
    | tee /etc/apt/preferences.d/99nginx
Package: *
 Pin: origin nginx.org
 Pin: release o=nginx
 Pin-Priority: 900

Updating repository information:

root@vm2406295323:~# apt update
Get:1 http://nginx.org/packages/ubuntu jammy InRelease [3590 B]
Get:2 http://nginx.org/packages/ubuntu jammy/nginx amd64 Packages [21.6 kB]
...
qwerty     2024-07-13 15:12:53

Installing NGINX:

root@vm2406295323:~# apt install nginx
...
The following NEW packages will be installed:
  nginx
...
Unpacking nginx (1.26.1-2~jammy) ...
Setting up nginx (1.26.1-2~jammy) ...
Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /lib/systemd/system/nginx.service.
...

Checking if NGINX installed:

root@vm2406295323:~# which nginx
/usr/sbin/nginx

Enabling NGINX service:

root@vm2406295323:~# systemctl enable --now nginx
Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable nginx

Checking NGINX service status:

root@vm2406295323:~# systemctl status nginx
● nginx.service - nginx - high performance web server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2024-07-13 12:57:49 MSK; 1min 6s ago
     ...
    Process: 2304 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=0/SUCCESS)
   Main PID: 2305 (nginx)
    ...
     CGroup: /system.slice/nginx.service
             ├─2305 "nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf"
             └─2306 "nginx: worker process" ...

At this point, we can visit http://89.111.132.97/ and see the NGINX welcome page.

Next step is installing PHP.

qwerty     2024-07-13 15:14:14

For PHP, I used these instructions. There may be good alternatives on English, or you can try Google translate this page linked above.

Installing prerequisites for PPA packages:

root@vm2406295323:~# apt install software-properties-common
...
The following additional packages will be installed:
  python3-software-properties
The following packages will be upgraded:
  python3-software-properties software-properties-common
...
Do you want to continue? [Y/n] y
...
Setting up python3-software-properties (0.99.22.9) ...
Setting up software-properties-common (0.99.22.9) ...
...

Installing PHP PPA package:

root@vm2406295323:~# add-apt-repository ppa:ondrej/php
...
Description:
Co-installable PHP versions: PHP 5.6, PHP 7.x, PHP 8.x and most requested extensions are included. Only Supported Ubuntu Releases (https://wiki.ubuntu.com/Releases) are provided.
...
CAVEATS:
1. If you are using php-gearman, you need to add ppa:ondrej/pkg-gearman
2. If you are using apache2, you are advised to add ppa:ondrej/apache2
3. If you are using nginx, you are advised to add ppa:ondrej/nginx-mainline or ppa:ondrej/nginx
...
Adding repository.
...

Installing NGINX PPA package (not sure we need it, just following the advice from previous output):

root@vm2406295323:~# add-apt-repository ppa:ondrej/nginx
...
Description:
This branch follows latest NGINX Stable packages compiled against latest OpenSSL for HTTP/2 and TLS 1.3 support.
...
Adding repository.

Updating repository information:

root@vm2406295323:~# apt-get update
...
Hit:6 https://ppa.launchpadcontent.net/ondrej/nginx/ubuntu jammy InRelease
Hit:7 https://ppa.launchpadcontent.net/ondrej/php/ubuntu jammy InRelease

Checking the latest version of PHP:

  1. Open the https://www.php.net/
  2. Check the text on button "What is new in x.y"
  3. If the text says "What is new in x.y", then x.y is the latest version.

Installing PHP (substitute x.y with values from previous step):

root@vm2406295323:~# apt-get install phpx.y
...
Creating config file /etc/php/8.3/mods-available/...
...
Setting up php8.3-cli (8.3.9-1+ubuntu22.04.1+deb.sury.org+1) ...
update-alternatives: using /usr/bin/php8.3 to provide /usr/bin/php (php) in auto mode
update-alternatives: using /usr/bin/phar8.3 to provide /usr/bin/phar (phar) in auto mode
update-alternatives: using /usr/bin/phar.phar8.3 to provide /usr/bin/phar.phar (phar.phar) in auto mode
...
Creating config file /etc/php/8.3/cli/php.ini with new version
...
Warning: Could not load Apache 2.4 maintainer script helper.
...

Installing PHP-FPM:

root@vm2406295323:~# apt-get install php8.3-fpm
...
The following NEW packages will be installed:
  php8.3-fpm
...
Setting up php8.3-fpm (8.3.9-1+ubuntu22.04.1+deb.sury.org+1) ...
...
Creating config file /etc/php/8.3/fpm/php.ini with new version
Created symlink /etc/systemd/system/multi-user.target.wants/php8.3-fpm.service → /lib/systemd/system/php8.3-fpm.service.
...

Enabling PHP-FPM service:

root@vm2406295323:~# systemctl enable --now php8.3-fpm.service
Synchronizing state of php8.3-fpm.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable php8.3-fpm
qwerty     2024-07-13 15:16:03

Now we want to test PHP installation, and all instructions I found online instruct to place info.php file in /var/www for testing purposes:

root@vm2406295323:~# echo '<?php phpinfo();' > /var/www/info.php
-bash: /var/www/info.php: No such file or directory

Oops! The directory not exists! Did I misconfigured something?

I created the "www" folder:

root@vm2406295323:~# mkdir /var/www
root@vm2406295323:~# echo '<?php phpinfo();' > /var/www/info.php
root@vm2406295323:~#

Now I moved to http://89.111.132.97/info.php and there is a 404 error. Indeed, my configuration is wrong.

In order to fix that issue, I tried steps from here, starting from section "Configure Nginx with PHP".

root@vm2406295323:~# nano /etc/nginx/sites-available/default
[ Directory '/etc/nginx/sites-available' does not exist ]

Oops, all these nonexistent directories and files gets scary! I hope that I don't go wrong with making these...

root@vm2406295323:~# mkdir /etc/nginx/sites-available
root@vm2406295323:~# nano /etc/nginx/sites-available/default

I created the basic server configuration file:

http {
    server {
    }
}

And entered Ctrl + X to exit the editor and save the file.

I also added /etc/nginx/sites-available to the list of directories where NGINX should look for configuration files.

Opened the NGINX configuration file:

root@vm2406295323:~# nano /etc/nginx/nginx.conf

And added the following line to the end of http context:

include /etc/nginx/sites-available/*;

Now I returned to default server configuration file and continued to edit the configuration:

root@vm2406295323:~# nano /etc/nginx/sites-available/default

In the server block, I added the following code:

listen 127.0.0.1:80;
root /var/www;

location / {
    index index.php index.html index.htm;
}

location ~ \.php$ {
    fastcgi_index index.php;
    fastcgi_pass unix:/run/php/php8.3-fpm.sock;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param SCRIPT_NAME $fastcgi_script_name;
}

Then I restarted the server:

systemctl restart nginx

And checked http://89.111.132.97/info.php and there is still a 404 error.

I even tried to test a simple html file (no PHP at all!):

root@vm2406295323:~# nano /var/www/index.html

And entered:

<html><head><title>test</title></head><body><h1>test</h1></body></html>

And tested http://89.111.132.97 and it still not working (only standard NGINX welcome page is displayed).

I opened the error log:

root@vm2406295323:~# cat /var/log/nginx/error.log

And there are the following lines:

2024/07/13 16:03:12 [error] 13749#13749: *1 open() "/usr/share/nginx/html/info.php" failed (2: No such file or directory), client: 46.151.156.96, server: localhost, request: "GET /info.php HTTP/1.1", host: "89.111.132.97"

Wait! Be we set up /var/www as root of our web site, why it is looking in /usr/share/nginx/html???

qwerty     2024-07-13 15:18:35

Let us grep through the server to find there /usr/share/nginx/html is set:

root@vm2406295323:~# grep -inr '/usr/share/nginx/html' /
/etc/nginx/conf.d/default.conf:8:        root   /usr/share/nginx/html;
/etc/nginx/conf.d/default.conf:18:        root   /usr/share/nginx/html;

Okay, let us look through that file as well:

root@vm2406295323:~# nano /etc/nginx/conf.d/default.conf

Yeah, this file contains the line:

root   /usr/share/nginx/html;

And this explains why it looks in /usr/share/nginx/html for files, though it not explains why configuration in /etc/nginx/sites-available/default is ignored.

I deleted this line and replaced the contents of the /etc/nginx/conf.d/default.conf file with the contents of /etc/nginx/sites-available/default file.

Then I deleted the /etc/nginx/sites-available folder because it was ignored anyway:

root@vm2406295323:~# rm -rf /etc/nginx/sites-available

I also removed this line from main NGINX configuration file, since it was ignored anyway:

include /etc/nginx/sites-available/*;

Then I restarted the server, and this time it failed to restart successfully:

root@vm2406295323:~# systemctl restart nginx
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.

And the status is:

root@vm2406295323:~# systemctl status nginx.service
× nginx.service - nginx - high performance web server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sat 2024-07-13 16:46:38 MSK; 39s ago
       ....

Jul 13 16:46:35 vm2406295323 systemd[1]: Starting nginx - high performance web server...
Jul 13 16:46:35 vm2406295323 nginx[14024]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)

According to https://stackoverflow.com/questions/14972792/nginx-nginx-emerg-bind-to-80-failed-98-address-already-in-use

I tried the line:

root@vm2406295323:~# pkill -f nginx & wait $!
[1] 14139
[1]+  Done                    pkill -f nginx
root@vm2406295323:~# systemctl start nginx

Okay, no errors this time, so I tried to visit http://89.111.132.97. And imagine what? It still shows me the NGINX default page!

qwerty     2024-07-13 15:20:22

I googled and googled and googled and finally found the trick: press Ctrl + F5 in browser and it will show you the actual content of the page, not the cached version.

However, http://89.111.132.97/info.php now returns "502 Bad Gateway". Still not working.

Let us see what we have in error.log:

2024/07/13 17:00:59 [crit] 14150#14150: *4 connect() to unix:/run/php/php8.3-fpm.sock failed (13: Permission denied) while connecting to upstream, client: 46.151.156.96, server: localhost, request: "GET /info.php HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.3-fpm.sock:", host: "89.111.132.97"

Well, maybe instead of unix:/run/php/php8.3-fpm.sock we must try some other line. Let us try this line instead:

fastcgi_pass localhost:9000;

It is not working too, accorging to error logs:

2024/07/13 17:15:45 [error] 14633#14633: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 46.151.156.96, server: localhost, request: "GET /info.php HTTP/1.1", upstream: "fastcgi://[::1]:9000", host: "89.111.132.97"

Well, let us try to work on /run/php/php8.3-fpm.sock solution again. This is one of the discussions I found on this topic:

https://stackoverflow.com/questions/23443398/nginx-error-connect-to-php5-fpm-sock-failed-13-permission-denied

According to that discussion, I added nginx to the www-data group:

root@vm2406295323:~# usermod -a -G www-data nginx

And returned the previous setting:

root@vm2406295323:~# nano /etc/nginx/conf.d/default.conf

Namely, the line:

fastcgi_pass unix:/run/php/php8.3-fpm.sock;

And restarted NGINX and PHP-FPM:

root@vm2406295323:~# systemctl restart nginx
root@vm2406295323:~# systemctl restart php8.3-fpm.service

Wow! The page http://89.111.132.97/info.php is working now!

Sorry I still not tested anything of CodeAbbey's site today. It was quite a work to do even a basic configuration of web site.

Rodion (admin)     2024-07-13 19:12:18
User avatar

Hi Qwerty!

If you don't mind, it would be better perhaps to create "issue" directly in the repository and put your notes there - as I doubt this thread is well suited for technical details.

Honestly I feel you are doing it the hardest possible way :) I don't know much about nginx (though it is used for sandbox for example) - but few minor things of this solution depend on Apache httpd as a web-server (I get used to think that nginx is more often met for proxy etc).

You could reuse the dockerfile in the repo for deploying the stuff on VPS (though of course it makes sense to extract database out of the container).

Also I'm unsure about the idea of using VPS/VDS for the website of such kind - as you'll need to solve a number of technical issues (you see, starting from installing the stuff - and ending with https certificates, ddos-protection etc), so I myself keep it on old-style shared hosting not without a reason. While again sandbox uses VPS.

That said, I learnt that our colleague Kevin (moxieman) had success with setting this up at awardspace (free shared hosting which we use for interactive problems by the way).

moxieman     2024-07-14 02:46:54
User avatar

Hi all!

Yes, I was able to find some success in deploying on Awardspace. It was relatively straightforward (though maybe a bit tedious) to upload the various files and set up the directories, and they also offer integrated mySQL for free accounts. Setting up the SQL database is only slightly more complicated, but isn't too bad once you get a feel for how the interface works. Overall it seemed a bit easier than the process qwerty is suffering through above...

Free accounts do expire after a year of inactivity though, so just be aware of that!

qwerty     2024-07-14 09:53:55

Hello everyone,

I finally installed and configured all the required software.

Then started with https://github.com/CodeAbbey/src/blob/v0.1-framework/index.php

I was really excited about the possibilty to learn from an experienced programmer's code, but after several hours of studying code in index.php realised that this is much above my level...

So, unlike Kevin aka moxieman, I was unable to deploy even a single file (index.php) to the site (Kevin, you are a real hero and the best progrmmer!).

Here are the moments I could not understand:

In the prepare function, we see:

addFile('conf.php');

What will happen if conf.php is missing? I do not see any special processing of false return value.

new \module\sys\Elems();

...do not bind the result to any value...

Why we create a class instance and then discard it? What is the purpose of the new \module\sys\Elems(); line?

$ctx = new \module\Context();
...
$ctx->elems->path = preg_replace('/^(.*\/)[^\/]*$/', '$1', $_SERVER['PHP_SELF']);

..but the class \module\Context does not contain the elems property.

Shouldn't that lead to error?

I made a test code:

<?php
$ctx = new \stdClass();
$ctx->elems->path = '';
echo $ctx->elems->path;

And after running it, I got:

Warning: Creating default object from empty value in the code at line 3

I tried to look into it a bit more and the class extends \module\sys\ProtoContext which have:

function __get($name) {
    $v = $this->names[$name] ?? null;
    if ($v !== null) return $v;
    $methodName = 'get' . ucfirst($name);
    if (!method_exists($this, $methodName)) {
        throw new \Exception("No property '$name' in Context!");
    }
    $res = $this->$methodName();
    if (is_object($res)) {
        $res->ctx = $this;
    }
    $this->names[$name] = $res;
    return $res;
}

I guess that we call:

protected function getElems() {
    return \module\sys\Elems::$elems;
}

When we are trying to set $ctx->elems->path - right?

Even if I am right, very complex way of doing things...

qwerty     2024-07-14 09:59:17

What is the meaning of this?

preg_replace('/^(.*\/)[^\/]*$/', '$1', $_SERVER['PHP_SELF']);

I am not good at regular expressions, can you add a PHP comment here to describe the purpose?

$page = isset($_GET['page']) ? $_GET['page'] : 'main';
$page = preg_replace('/[^a-z0-9\_]/', '', $page);
$page = str_replace('_', '/', $page);

I tested the original CodeAbbey site and did not found where 'page' is used in the request. Can you give an example?

Wouldn't there be any problems that addfile function is declared with all lowercase letters but called with addFile and addfile?

Or PHP is a case insesitive language (sorry for maybe silly question, I am not famillar with the language)?

In the addfile function...

function addfile($name) {
    global $model, $ctx;
    if (!file_exists($name)) {
        return false;
    }
    include($name);
    return true;
}

...I cannot understand the global $model, $ctx; here, as there are no usages. But I already understood that PHP is very complex language, so maybe some subtle side effects of these declarations?

$r = addfile($ctlFile);

Can anyone explain the assignment above? I do not see any usage of $r...

addFile("ctl/_hookPreRender.php");

I did not found _hookPreRender.php.

array_push($ctx->elems->styles, $ctx->elems->page);
array_push($ctx->elems->scripts, $ctx->elems->page);

I do not get how a page can serve itself both as stylesheet and as a script? Can anyone elaborate a bit?

What is this?

ob_start();
...
$ctx->elems->contentResult = ob_get_clean();

I tried to read:

https://www.php.net/manual/ru/function.ob-start.php

But did not understand much...

addFile("ctl/_hookAfter.php");

I did not found _hookAfter.php.

What is the func_num_args in the url function?

Does that mean that url function can accept some arguments not described in the function definition?

What is these arguments and why not explicitly add these arguments to the definition to make programmer's life easier?

} else if ($numArgs > 3) {
    $i = 1;
    $args = func_get_args();
    $query = array();
    while ($i < $numArgs - 1) {
        if ($args[$i] !== null && $args[$i + 1] !== null) {
            array_push($query, $args[$i] . "=" . $args[$i + 1]);
        }
        $i += 2;
    }
    $res .= ($rewrite ? '?' : '&') . implode('&', $query);
}

This is not easier than the code above? https://www.php.net/manual/en/function.http-build-query.php

function aurl($url)

It is not clear from the name what is difference between url and aurl functions, maybe choose another name to help me understand...

spl_autoload_register(function($class) {
    $path = str_replace('\\', '/', $class) . '.php';
    if ((@include $path) === false) {
        echo "Loading $class file failed!\n";
    } elseif (!class_exists($class, false) && !interface_exists($class, false)) {
        echo "Loading $class failed!\n";
    }
});

I do not understand to where errors are echoed. Directly to the page shown to user?

Sorry it seems that I won't get the site running for a while, maybe for 3-4 months.

I am just an unexperienced and very low-skilled programmer, it took me a good five hours to read the index.php file, so you can imagine how much time it will take for me to read all files in the reposiory...

Thankfully, Kevin seems to be much more experienced programmer, so he somehow handled all these issues - amazing work!

Rodion (admin)     2024-07-14 18:18:41
User avatar

Qwerty, Hi! first of all, don't regard it as the "code by experienced user". I don't have PHP experience besides this site and it was mostly written 10 years ago when I had much less experience in anything at all.

Hopefully we'll improve and clarify it by and by, but generally it is not great example for study.

Again I recommend to ask questions about code using GH issues, it's more handy than here.

Deploy all the files of the step 1 and see if you can make it work. Much understanding of the code mainly is needed if you nead to tweak it.

aulr is for abs.url, rarely used. Elems set static field in constructor (a bit confusing, yes). addFile allows file to be missing, so it is ok that some files are not there yet. error handling is very poor. you may in some cases need to look for them in page source (ctrl-u in browser).

I'll return with other answers, hopefully in gh issues, but note that normally one is not supposed to have deep knowledge of the language to use this (otherwise it will be useless). I think Kevin has just basic knowledge of PHP and I myself won't recollect some things without googling :)

moxieman     2024-07-15 13:55:55
User avatar

Hi qwerty! I would agree with Rodion - I'm hardly an experienced PHP programmer! I wrote my first line of PHP less than five months ago to solve the Dynamic Web Page problem. I've had a little PHP practice in writing checkers for problem submissions and also playing around with SQL interfacing, but these files are the first time I'm seeing a lot of "real" web dev concepts (login system, js integration, global PHP objects, etc).

Luckily as Rodion mentioned, "knowing how it works" isn't really required for "getting it to run". I'd recommend just uploading the files and getting it working "like magic" first, then start playing around with lines of code to see what the effects are. Thankfully everything is already set up to work "as-is" if you just download all the files directly from the GitHub repository then upload to the hosting server (except for very simple tweaks of conf.php in the later branches to define the SQL login credentials).

I certainly couldn't explain very well what every line of index.php is doing, but I'm slowly working my way through it and trying to figure it out as I go. So don't worry if much of the code details seem quite confusing at this point! I empathise with wanting to understand how everything is working from the start, but I've never met anyone who learned how to work on car engines without first having much experience driving :)

I can try to answer some of your PHP-specific questions...

  • PHP is a "partially" case-sensitive language. Variable names are case-sensitive, but function names are not. So $myvar and $MyVar are unique, but myfxn() and MyFxn() are the same.

  • Regarding those 4 regexp lines...

    • I think the 1st one is extracting the filepath of the running script (index.php in this case),
    • the 2nd one is checking if the url is directing to some specified page and defaulting to main otherwise, and
    • the 3rd/4th lines are santizing the filepath string to lowercase and acceptable characters.

However note that looking into this raises further questions for me (santizing url .../iNdEx/TaSk_LiSt to .../index/task_list works for my mirror site but apparently not the main site? wouldn't the underscore in task_list get converted to a forward slash? is that even what those lines are really doing?) but the main takeaway here should be that those questions don't need to be answered to run the site, and now that I have a "partially-magic" running site to play with I could probably answer those questions by taking some time, commenting out lines of code, and generally playing with things to see their effects :)

Rodion (admin)     2024-07-15 16:36:01
User avatar

Variable names are case-sensitive, but function names are not

Funny. I never know until today :)

BTW, Kevin, if you would like, I also recommend to put questions into issues in GH - just right now it looks we are collecting more and more of them and answering some of them requires some "archeology" work in the code - despite the "core" or framework is pretty small (hopefully) I definitely don't remember how some of these things evolved :)

About ob_start. See, normally PHP page works like HTML with insertions of small scripting elements (in php tags). Of course it long since changed and particularly here we use the full-blown script or application in PHP code which needs at some point to load other PHP which mainly contains HTML (e.g. page templates), decorate it with some "layout", insert some "fragments" and only then send to output. Regretfully when we include some file, it is sent to output immediately, so we use this queer magic - ob_start instructs interpreter to stop sending things into output, but other put them to some "output buffer" (I guess hense the prefix "ob"). When the template is thus "collected" we get it from the buffer and process at our will. This is definitely confusing because some errors in the code before this may be also sent to output, breaking the page, headers and everything.

qwerty     2024-07-16 08:35:44

Kevin, Rodion, first of all I want to thank you for great answers! Learned a lot from these.

Second, I understand that a GitHub issue is a better place to continue discussion, but I am not an experinced GitHub user yet (my only experinece is that python-django codeabbey repository linked above). Perhaps I will try to make my first issue with more free time in disposal.

For now, the following was done:

  1. Cloned the latest version repository to my local computer, so I can make any needed changes in the code before uploading it to site.

    git clone https://github.com/CodeAbbey/src.git

  2. Not making any changes in the code yet, installed WinSCP and transfered all the code to /var/www folder of the server.

  3. Of course, this did not work right away: when I tried to load http://89.111.132.97/index.php, the message appeared: "Loading mysqli file failed!"

  4. I checked README.md, navigated to http://89.111.132.97/sqlexec.php and inserted the contents of dbinit.sql into text field (I did not make any changes to the sql)

  5. I got: Result: fail (127) sh: 1: mariadb: not found.

  6. I ran apt update then apt install mariadb-server.

  7. Tried to execute SQL again and got: Result: fail (1) ERROR 1698 (28000): Access denied for user: 'www-data'@'localhost'

This time I do not fully understand the error, can anyone give me hints on how should I debug this error?

qwerty     2024-07-16 08:45:36

By the way, www-data is the user who runs PHP code on the server.

qwerty     2024-07-16 09:14:18

I checked the code in the sqlexec.php and understood that it runs a quite simple command.

So I just copied the dbinit.php file to the home folder of root user and run:

cat dbinit.php | mariadb -B

This worked like a charm, the database is created as well as all tables.

And I fixed the error: "Loading mysqli file failed!" by running the command:

apt-get install php8.3-mysqli

However, now I'm getting a HTTP 500 error at http://89.111.132.97/index.php.

The NGINX error log says:

Uncaught mysqli_sql_exception: You have an error in your SQL syntax: ... near the ')' at line 1 in /var/www/module/dao/MysqlDao.php:67
Rodion (admin)     2024-07-16 10:39:13
User avatar

Qwerty, Hi!

As about github issues - just register on GH, navigate to the project page, click "issues" and then green button "New Issue". the same markdown format is used, but with some additional things like cross-referencing issues, linking code etc. If the site is down it will be difficult to you to find all these notes if they are hosted on the site itself :)

Installing mariadb and workaround for the user to run mysql with sqlexec could be found in Dockerfile in the root folder. Even though you decided not to use docker, there are all the commands I myself used to install things so it may be helpful :)

Error itself looks queer, as you understand without some more debugging efforts on your side it is hard to tell what may be wrong. Open this file at this specific line, wrap it with try ... catch (you'll easily google for syntax - I don't remember myself) - and log the query if there is error perhaps. Or you can enable query logs in mysql itself.

BTW the "latest version" may contain some intermediate changes... "step-X" branches should be more stable, I think.

Have you configured anything about DB in conf.php? Or defaults worked for you?

Ah, also you can probably change error reporting with the first line in conf.php (one with "display_errors") - though still you may need to look into page source after loading to find them.

Rodion (admin)     2024-07-18 09:33:27
User avatar

Hi, the issue was due to recent changes reacting to empty task list (so really master branch should be used with caution)

Either update files (there are fixes in master now) or populate db with queries from the taskinit file.

particular error was due to SQL clause ...id in ($task_id_list)... which unexpectedly is "incorrect syntax" in mysql when parentheses are empty.

qwerty     2024-07-18 12:38:59

Hello, Rodion,

Just uploaded the latest version of sources to server, and everything works! Hurrah!

By the way, I tried to debug the issue yesterday, but with no avail.

Here is how I tried to debug:

function find($cond = null, $limit = null, $offset = null) {
    // Added the line below after reading
    // https://stackoverflow.com/questions/27345377/try-catch-in-mysqli
    mysqli_report(MYSQLI_REPORT_STRICT | MYSQLI_REPORT_ALL);

    $query = "select * from {$this->table}";
    $query .= $this->clauses($cond, $limit, $offset);
    try {
        $res = $this->conn->query($query);
    }
    catch (mysqli_sql_exception $e) {
        fwrite(
            STDERR,
            "In /var/www/module/dao/MysqlDao.php:find. Query is: $query\n"
        );
        file_put_contents(
            "/var/www/crash.log",
            "In /var/www/module/dao/MysqlDao.php:find. Query is: $query\n"
        );
        throw $e;
    }
    return $this->objectsArray($res);

    // Other topics that did not helped me (but may give you an idea):
    // https://www.php.net/manual/ru/mysqli-driver.report-mode.php
    // https://stackoverflow.com/questions/72241551/why-mysqli-sql-exception-is-not-caught-by-php
}

But PHP just kept ignoring all error handling: no messages to stderr, nothing written in crash.log.

After several hours trying to properly set up the error handling, I finally gave up. In Python it was much easier to handle errors...

If anyone knows what was wrong with the above code, I would be grateful to know.

Rodion (admin)     2024-07-20 08:18:32
User avatar

Hi Qwerty!

I'm not that deep in PHP so it's the first time I hear of specific mysqli_sql_exception :) My code is simpler:

try {
    $res = $this->conn->query($query);
} catch (\Exception $e) {
    echo "Exception on query $query";
}

it may be good to set $res = []; probably after error is caught to prevent further errors. anyway it works for me and you can test it by temporarily breaking the query in ($taskIds) in the ctl/Main.php for example (e.g. remove variable leaving empty parentheses).

I'm not sure if there could be some server-side options for PHP preventing exception handling (for errors they exist, see above).

Somewhat important update - step 8 - here we start taking care of customization needs - e.g. if someone of you deploy your own version of the site, you want to change its name, front page elements etc - for this we'll be providing dedicated settings etc (so that many important changes could be done in a single place and without messing in the sources).

Rodion (admin)     2024-07-21 17:37:56
User avatar

Added steps 9 and 10 (task tags and wiki pages)...

and more important:

Recorded Video Instruction of deploy to AwardSpace - so that now anyone should be able to clone and setup the site in just half an hour - this doesn't require special knowledge, but there are few subtle points (I stumbled upon them myself).

qwerty     2024-07-27 10:18:58

Hello, Rodion,

Good thing, it is a weekend again!

What should I do today, after deployment the site? As far as I can remember, you opensourced CodeAbbey in order for us to help finding vulnerabilities, right?

What exact vulnerability I should seek? I am by no way a security specialist, so please forgive me my silly question.

Please login and solve 5 problems to be able to post at forum