文件列表:

application/ (0, 2023-10-09)
application/.htaccess (123, 2023-10-09)
application/cache/ (0, 2023-10-09)
application/cache/index.html (131, 2023-10-09)
application/config/ (0, 2023-10-09)
application/config/autoload.php (4025, 2023-10-09)
application/config/config.php (21680, 2023-10-09)
application/config/constants.php (4322, 2023-10-09)
application/config/database.php (4899, 2023-10-09)
application/config/doctypes.php (2441, 2023-10-09)
application/config/foreign_chars.php (3046, 2023-10-09)
application/config/hooks.php (417, 2023-10-09)
application/config/index.html (131, 2023-10-09)
application/config/memcached.php (498, 2023-10-09)
application/config/migration.php (3032, 2023-10-09)
application/config/mimes.php (10233, 2023-10-09)
application/config/per_method_limits.php (972, 2023-10-09)
application/config/profiler.php (477, 2023-10-09)
application/config/rest.php (20035, 2023-10-09)
application/config/routes.php (1985, 2023-10-09)
application/config/smileys.php (3181, 2023-10-09)
application/config/user_agents.php (6129, 2023-10-09)
application/controllers/ (0, 2023-10-09)
application/controllers/Restapi.php (14046, 2023-10-09)
application/controllers/index.html (131, 2023-10-09)
application/core/ (0, 2023-10-09)
application/core/index.html (131, 2023-10-09)
application/helpers/ (0, 2023-10-09)
application/helpers/index.html (131, 2023-10-09)
application/hooks/ (0, 2023-10-09)
application/hooks/index.html (131, 2023-10-09)
application/index.html (131, 2023-10-09)
application/language/ (0, 2023-10-09)
application/language/english/ (0, 2023-10-09)
application/language/english/index.html (131, 2023-10-09)
application/language/english/rest_controller_lang.php (911, 2023-10-09)
application/language/index.html (131, 2023-10-09)
application/libraries/ (0, 2023-10-09)
... ...

# JOBE Version: 1.8.0, 9 October 2023 Author: Richard Lobb, University of Canterbury, New Zealand Contributors: Tim Hunt, Fedor Lyanguzov, Kai-Cheung Leung, Marcus Klang ## Introduction Jobe (short for Job Engine) is a server that supports running of small compile-and-run jobs in a variety of programming languages. It was developed as a remote sandbox for use by [CodeRunner](http://github.com/trampgeek/coderunner), a Moodle question-type plugin that asks students to write code to some relatively simple specification. However, Jobe servers could be useful in a variety of other contexts, particularly in education. A job specifies a programming language, the source code, the standard input to the run and an optional list of additional files. Jobe compiles the source code (if compilation is appropriate in the specified language) and runs it with the given input data. It returns a run_result object containing various status information plus the output and error output from the run. The interface is via a RESTful API, that is documented [here](./restapi.pdf). The languages C, C++, Python3, Python2, Octave, Java, Pascal and PHP are all built-in. Other languages can be added fairly easily although if using Jobe from CodeRunner it is usually even easier to write a Python-based question type that scripts the execution of the required language. See the [CodeRunner documentation](http://coderunner.org.nz/mod/book/view.php?id=193&chapterid=749) for an example. The Computer Science quiz server at the University of Canterbury switched to exclusive use of the Jobe sandbox in early July 2014. Since then it has run many hundreds of thousands of Python3, C, Java and Octave jobs unattended with only a few minor bug fixes and security refinements. ## Implementation status The current version of Jobe (Version 1.7.1, May 2023) implements a subset of the originally documented API, sufficient for use by CodeRunner. It has been used for many years at the University of Canterbury for several years, running many millions of submissions. Jobe is also used by over 600 other CodeRunner sites around the world. It can be considered stable and secure, though it should be run only on a separate appropriately-firewalled server. With reference to the original API spec, only immediate-mode runs are supported, with run results being returned with the response to the POST of the run requests. Run results are not retained by the server (unless *run\_spec.debug* is true; see the API), so *get\_run\_status* always returns 404 not found. File PUTs are supported but not POSTs. When used by CodeRunner, file IDs are MD5 checksums of the file contents. Since version 1.6, the Jobe server cleans the file cache whenever available disk space drops below 5% of the disk size. It simply deletes all files that haven't been used for 2 days or more, so the server must have enough free disk space to stay below 95% full for at least two whole days of running. For CodeRunner clients this should not be a problem unless question authors enable large classes of students to attach large files to their submissions. Support files attached by question authors are unlikely to be a problem; a Jobe server at the University of Canterbury serving a large Moodle client with many thousands of questions accumulated only 200 MB of support files over several years. For sandboxing, Jobe uses the [domjudge](http://domjudge.org) *runguard* program to run student jobs with restrictions on resource allocation (memory, processes, cpu time) as a low-privileged user. However it does not restrict any system calls. Programs may write binary output but the results are returned to the caller JSON-encoded, which requires UTF-8 strings. To avoid crashing the json-encoder, the standard output and standard error output from the program are checked to see if they're valid utf-8. If so, they're returned unchanged. Otherwise, they're taken as 8-bit character streams; characters below '\x20' (the space character) and above '\x7E' are replaced by C-style hexadecimal encodings (e.g. '\x8E') except for newlines which are passed through directly, and tabs and returns which are replaced with '\t' and '\r' respectively. If Jobe is to correctly handle utf-8 output from programs, the Apache LANG environment variable must be set to a UTF-8 compatible value. See the section *Setting the locale* below. Jobe is implemented using Ellis Lab's [codeigniter](http://codeigniter.com) plus the [RESTserver plugin](https://github.com/chriskacerguis/codeigniter-restserver) originally written by Phil Sturgeon and now maintained by Chris Kacerguis. ## Installation **WARNING** Jobe is primarily intended for use on a server that is firewalled to allow connections from authorised client machines only. If you install it on a machine without such firewalling, and do not control access with API keys (see later), anyone will be able to connect to your machine and run their own code on it! **CAVEAT EMPTOR!** NOTE: a video walkthrough of the process of setting up a Jobe server on a DigitalOcean droplet is [here](https://www.youtube.com/watch?v=dGpnQpLnERw). Installation on Ubuntu 22.04 systems should be straightforward but installation on other flavours of Linux or on systems with non-standard configurations may require Linux administrator skills. An alternative approach, and probably the simplest way to get up and running, is to use the [JobeInABox](https://hub.docker.com/r/trampgeek/jobeinabox/) Docker image, which should be runnable with a single terminal command on any Linux system that has docker installed. Thanks to David Bowes for the initial work on this. Please be aware that while this Docker image has been around for a couple of years the developer has not himself used it in a production environment. Feedback is welcomed. The steps to fire up a Jobe Server on Digital Ocean using JobeInAbox are given below in section *Setting up a JobeInAbox Digital Ocean server*. However, for security and performance reasons it it *strongly* recommended to run Jobe on a dedicated server, even when running it in a container. Jobe runs only on Linux, which must have the Apache web server installed and running. PHP must have been compiled with the System V Semaphone and shared-memory functions enabled (see here)[http://www.php.net/manual/en/sem.setup.php], but that's the norm. Access Control Lists (ACLs) must be enabled; they normally are but if the `/home/jobe` directory lands up on a mounted volume, you may need to explicitly enable ACLs in the `mount` command or in `/etc/fstab`. The Python3 and the C development system must also be installed. ### Installing the necessary dependencies On Ubuntu-22.04, the commands to set up all the necessary web tools plus all currently-supported languages is the following: sudo apt-get install apache2 php libapache2-mod-php php-cli\ php-mbstring nodejs git python3 build-essential default-jdk\ python3-pip fp-compiler acl sudo sqlite3 sudo apt-get install --no-install-suggests --no-install-recommends octave Octave and fp are required only if you need to run Octave or Pascal programs respectively. If you wish to use API-authentication, which is generally pointless when setting up a private Jobe server, you also need the following: sudo apt install mysql-server php-mysql Similar commands should work on other Debian-based Linux distributions, although some differences are inevitable (e.g.: acl is preinstalled in Ubuntu, whereas in debian it must be installed). A Raspberry Pi user reports that they additionally had to use the command apt-get install --fix-missing which may help with broken installs on other systems, too. ### Setting up pylint (if you want it) Firstly, install pylint for your required version of python (assumed here to be python3) with the command: sudo -H python3 -m pip install pylint You also need to build the /etc/pylintrc file to set the default options with one of the following commands, which must be run as root (don't just try prefixing the command with sudo, as the output redirection will fail). Firstly try the command: pylint --reports=no --score=n --generate-rcfile > /etc/pylintrc If that gives you an error "no such option: --score" (which happens with older versions of pylint), try instead pylint --reports=no --generate-rcfile > /etc/pylintrc ### Installing Jobe Clone the Jobe project in the web root directory WEBROOT (usually /var/www/html). Do not clone it elsewhere and attempt to add it to web root with symbolic links. That breaks this installer. In what follows, replace WEBROOT with either /var/www or /var/www/html as appropriate. To clone Jobe: cd WEBROOT sudo git clone https://github.com/trampgeek/jobe.git Installation is performed by the install script, which must be run as root so that it can add the required jobe run users (jobe00, jobe01, etc) and set-up a jobe-sudoers file in /etc/sudoers.d that allows the web server to execute the runguard program as root and to kill any residual jobe processes from the run. Before running the install script, you might wish to edit the file /var/www/html/jobe/application/config/config.php Find the line $config['jobe_max_users'] = 8; and decide if that value, which sets the maximum number of jobs that can be run at once, is appropriate for your hardware set up. A rule of thumb is to set this to the number of cores on your machine, but if you plan on running lots of partially-I/O-bound jobs, you could consider larger numbers. Having set that value to your satisfaction: cd WEBROOT/jobe sudo ./install On Centos6 systems (and possibly early Centos7 releases) you should also comment out the line Defaults requiretty in /etc/sudoers. This was (reported as a bug)[https://bugzilla.redhat.com/show_bug.cgi?id=1196451] and was fixed in later RHEL releases. ### Setting the locale By default, Apache is configured to use the C locale. This means that programs generating, say, UTF-8 output will fail with an error UnicodeEncodeError: 'ascii' codec can't encode character ... If you wish to run UTF-8 code (recommended) you should find the line in the Apache envvars file (on Ubuntu systems this is to be found at /etc/apache2/envvars) export LANG=C and change it to either C.UTF-8 (which changes the charset to UTF-8 but leaves other locale settings unchanged) or to the required standard locale value, e.g. export LANG=en_NZ.UTF-8 Make sure that whatever locale you use is installed on the Jobe server. Then restart apache with the command sudo service apache2 restart Note: 1. The comment in the Apache envvars file suggesting the use of the default locale probably won't work, as this will also just give you ASCII text. 2. To take advantage of the UTF-8 capabilities in CodeRunner you will need to use Version 3.3 or later. ## Setting up a JobeInAbox Digital Ocean server For people wanting to get a Jobe server up in hurry, the following is probably the simplest approach. This uses a minimal Digital Ocean virtual machine, to run the Docker *JobeInAbox* image; you should increase memory and core for production servers. Other cloud servers, such as Amazon ECS, can of course also be used. 1. Set yourself up with an account on [Digital Ocean](https://cloud.digitalocean.com). 2. Create new Droplet: Ubuntu 22.04. x64, minimal config (1GB CPI, 25GB disk) 3. Connect to the server with an SSH client. 4. Install docker (see https://phoenixnap.com/kb/how-to-install-docker-on-ubuntu-18-04): sudo apt update; sudo apt install docker.io 5. Launch JobeInABox with Docker: sudo docker run -d -p 80:80 --name jobe trampgeek/jobeinabox:latest At this point you have a running Jobe server. You can check it's working by browsing to http:///jobe/index.php/restapi/languages You should get presented with a JSON list of installed languages. And you can connect your CodeRunner plugin to it by setting the new JobeServer IP number in the Admin panel of the plugin. You're in business! All that remains is to firewall your new server so that only your Moodle server can use it, and so it can't itself open outgoing connections. For example: sudo apt install ufw sudo ufw default reject outgoing sudo sudo ufw allow in 22/tcp sudo ufw allow in proto tcp to any port 80 from sudo ufw enable ## Testing the install The program *testsubmit.py* allows you to test your jobe server. For information on how to use it run this command on the Jobe server: python3 testsubmit.py --help ### Testing general correctness To test the installation, first try running the tester with the command python3 testsubmit.py The first time you run this command, the initial step of obtaining all the different versions of all language is slow, as it has to test-drive all compilers and interpreters. Be patient. Results are cached in a file in /tmp so subsequent runs will be much faster, at least until the next reboot, when the list is rebuilt. All going well, you should then be able to copy the *testsubmit.py* file to any client machine that is allowed to access the jobe server and rerun the command with a line of the form python3 testsubmit.py --host='jobe.somehow.somewhere' --port=4000 where the host and port are set to reference the jobe server. ### Testing performance To test the performance of your new Jobe server, run the testsubmit program on your client machine again, this time with a --perf command line argument, e.g. python3 testsubmit.py --perf --host='jobe.somehow.somewhere' --port=4000 The test will print information on the maximum burst of C compile-and-run submissions the server can handle in isolation and the sustained rate of submissions over a 30 second window by default. The figures you get are upper bounds, since the program being used for testing is a minimal 'hello world' program. It's also possible that the Moodle server cannot deliver jobs to the Jobe server at the maximum rate. Languages like C, PHP and nodejs have similar performance since the communication and server overheads dominate the performance. For slower languages like C++ and particularly Java however, you will much lower throughput. To test Java, for example, type add the argument 'java' to the above command, i.e. python3 testsubmit.py --perf --host='jobe.somehow.somewhere' --port=4000 java *WARNING*: you should not run the performance test on a live production server, as it repeatedly pushes the server into overload, which will result in other users' jobs also receiving server-overload errors. ## Using Jobe Usually Jobe is used as a server for Moodle CodeRunner questions. So once jobe has been installed and tested with `testsubmit.py` it can be used by CodeRunner questions by plugging the Jobe server hostname into the CodeRunner administrator settings, replacing the default value of `jobe2.cosc.canterbury.ac.nz`. However, Jobe can also be used standalone. The `simpletest.py` program shows how it can be invoked from a Python client. Note that the POST request payload must a JSON object with a *run_spec* attribute as specified in the document *restapi.pdf*. For example, the following POST data runs the classic C "Hello World" program: {"run_spec": {"language_id": "c", "sourcefilename": "test.c", "sourcecode": "\n#include \n\nint main() {\n printf(\"Hello world\\n\");\n}\n"}} The POST request must have the header Content-type: application/json; charset-utf-8 and should be sent to a URL like localhost/jobe/index.php/restapi/runs For example, the following Linux `curl` command runs the C Hello World program: curl -d '{"run_spec": {"language_id": "c", "sourcefilename": "test.c", "sourcecode": "\n#include \n\nint main() {\n printf(\"Hello world\\n\");\n}\n"}}' -H "Content-type: application/json; charset-utf-8" localhost/jobe/index.php/restapi/runs ## Updating Jobe If you wish to update an existing version of Jobe to a new one, first put the the client Moodle server into maintenance mode. Reboot the Jobe server. Then `cd` into the Jobe directory, do a `git pull` to update the code, then run the installer with the --purge option, i.e. sudo ./install --purge Check that all is well by testing as in the section "Testing the install" above. Lastly take the Moodle server out of maintenance mode again. ## Debugging If you have problems installing Jobe, here are some things to check. If the install script fails, check the error message. You should be able to read through the script and figure out what went wrong. Otherwise ... 1. Check the install went OK: 1. Make sure your webserver has read access to the entire jobe subtree. 1. Make sure your webserver has write access to jobe/files 1. Make sure there exist users jobe and jobe00 through jobe09. 1. Make sure there is a directory /home/jobe/runs owned by jobe and writeable by the webserver. It should not be readable or writeable by all. 1. Make sure there is a directory /var/log/jobe. If the install appears OK but testsubmit.py fails: 1. If you get messages "Bad result object", something is fundamentally broken. Start by rebooting your server, and make sure Apache is running, e.g. by browsing to http://\. 1. Try pointing your browser at http://\/jobe/index.php/restapi/languages This should return a JSON list of languages. If not, you may at least get a readable error message. 1. You are running testsubmit.py with Python3, right? 1. Check the apache error log. 1. If something unexpected happened with the actual run of a program, find the run in /home/jobe/runs and try executing the program manually. [The run directory contains the source file, the bash command used to run it, plus the compile output and (if it ran) the stderr and stdout outputs. 1. Check for any error messages in /var/log/jobe/*. 1. Turn on debug level of logging in jobe/application/config/config.php by setting the log_threshold to 2 (around line 183). You should now get screeds of log info in the directory /var/log/jobe. Most of this comes from the framework; look for lines beginning *jobe*. These are all issued by restapi.php in application/controllers, which is the top level handler for all http requests. 1. If you are getting Overloaded errors, then you can display the in-memory locks on the Jobe users with this PHP one-liner: ```php -r 'print_r(shm_get_var(shm_attach(ftok ("/var/www/html/jobe/application/libraries/LanguageTask.php", "j")), 1));``` If you still can't figure it out, email me (Richard Lobb; my gmail name is trampgeek). ## An optional extra installation step [For paranoid sysadmins only]. Submitted jobs can generally write files only into the temporary directory created for their run within the '/home/jobe/runs' directory. Exceptions to this rule are the /tmp, /var/tmp, /var/crash and /run/lock directories all of which conventionally can be written into by any Linux process. The temporary working directory and any files in the writable directories mentioned above are deleted on the termination of the run. However, depending on the size of the various partitions and the allowed maximum run time, it might in principle be possible for a rogue process, or a deliberate attacker, to run the system out of disk space in a particular partition (probably /tmp, which is us ... ...

近期下载者：

相关文件：

评论：[我要评论] [举报此文件]

收藏者：