offliner

Offliner is a tool to make a website offline viewable. It's a concurrent web crawler which crawls a website and saves all the pages and static files in a directory.
It can use both, multi-processing & multi-threading as its concurrency model. In addition to those original concurrency modes, serial execution is available. When static-file downloading is enabled, supported CSS, JavaScript, and image files are saved and matching links in saved pages are rewritten.

• Features
• Requirements and build
• Usage
• Examples
• Output layout
• Documentation
• Known limitations
• Todo
• License

• Serial scraping.
• Multi-threaded scraping.
• Multi-process scraping.
• Save static files (css, js, img).
• Edit the links on the pages to reference the local files.
• The multi-threaded mode uses goroutines; the multi-process mode uses worker programs and Unix-domain sockets.
• Static-file handling supports CSS, JavaScript, JPG, PNG, and JPEG references when enabled.
• Limit the number of pages and concurrent execution units.

The project uses Go modules and declares Go 1.17. Build the main command from the repository root:

go build -o offliner ./cmd

The multi-process option expects a process worker executable in the current working directory, alongside offliner. Build it from the repository root with:

go build -o process ./process

The multi-process implementation uses Unix-domain sockets under /tmp; use that mode on a Unix-like environment. The other modes do not require the worker executable.

You need to provide a full URL to start the scraping. You can use the defined flags to control the features. If you intend to use the multi-process form, the "process" program must exist in the same directory as the "offliner" program. Run the program from a working directory where it can create the relative ../output/ directory.

-h     show help.
-url   full URL of the start page.
-f     save static files too. It also edits the pages so the links reference the local files.
-a     use multi-processing instead of multi-threading as the concurrency model.
-n     maximum number of the pages to be saved. (default is 100)
-p     maximum number of the execution units (goroutines or processes) to run at the same time. (default is 50)
-s     run the scraper in a non-concurrent (serial) fashion.

-s selects serial execution; otherwise the default is goroutine-based execution. -a selects the multi-process implementation when serial mode is not selected. The flags are parsed by Go's standard flag package, so values can be supplied as -url=https://example.com or as -url https://example.com.

Multi-threaded scraping. Save max 100 pages using max 90 goroutines. Save static files too.

./offliner -url=https://urmia.ac.ir -n=100 -p=90 -f

Multi-process scraping. Save max 100 pages using max 50 processes.

./offliner -url=https://urmia.ac.ir -n=100 -p=50 -a

Serial scraping. Save max 100 pages. Save static files too.

./offliner -url=https://urmia.ac.ir -n=100 -s -f

The program creates output paths relative to its current working directory. For a starting host such as example.com, the output is organized as follows:

../output/example.com/
├── pages/       downloaded HTML pages
└── static/
    ├── css/     downloaded CSS files
    ├── js/      downloaded JavaScript files
    └── img/     downloaded image files

Page filenames are derived from their URLs. Static files are downloaded only when -f is enabled, and the current implementation recognizes CSS, JavaScript, JPG, PNG, and JPEG references.

Project documentation is organized in docs/. The system overview is embedded below and is also available as a standalone SVG or HTML diagram.

• System architecture, including the architecture diagram and runtime details.
• Usage and crawl behavior.
• Development and repository map.

• The crawler follows same-host HTML links and skips links with query strings; it does not crawl arbitrary external hosts.
• Static-file handling is limited to the link, script, and image tags and file extensions described above. It does not rewrite every possible asset reference, such as CSS url(...) values.
• The multi-process path uses a separate worker implementation. Its page processing and link handling differ from the serial and goroutine paths, and static-file downloading is not implemented in that worker.
• Output directories and worker executable lookup are relative to the current working directory.

See the architecture notes and usage notes for more detail on execution paths and scope.

• Improve multi-processing design.
• Add a logger.
• Make the scraper a separate package (library).

GNU General Public License v3.0