All about the development of Exponential (CMS) by Graham Brookins, 7x (formerly Brookins Consulting) and our think tank kracker.org.
Hello Exponential Community,
7x keeps the complete doc.ez.no documentation as a static HTML archive in a public GitHub repository. Two free Exponential extensions turn that archive, or any folder of HTML or Markdown, or a live website, into real content objects in your own Exponential site with one command. One run gives finished work; running it again is never needed, but always safe. The result runs live at doc.exponential.earth.
The repositories
- se7enxweb/exponential: Exponential 6.0.15 or later, the CMS.
- se7enxweb/exponential.doc.exponential.earth: the static archive of doc.ez.no, 88,680 pages.
- se7enxweb/exp_static_html_imports: the import framework. Four source types, your own types, safe re-runs, reports.
- se7enxweb/exp_doc: documentation content classes, the import rules for the archive, and two designs: Classic (doc2025, the static site exactly) and Modern (doc2026).
All are free software under the GNU General Public License v2.0 (or any later version).
What you need
- Exponential 6.0.15 or later (earlier 6.0 versions work without the audit events and the console aliases).
- PHP 7.4 or later; PHP 8.0 to 8.5 are supported.
- Any of five databases: MySQL/MariaDB, PostgreSQL, SQLite, Oracle or MongoDB.
- The online editor ezoe, active by default. Its HTML parser turns pages into rich text.
- git on the server, for sources given as a repository.
- For exp_doc also: eztags (labels) and xrowmetadata (the metadata attribute).
Example 1: a folder of Markdown, in minutes
Install the framework from your Exponential root:
git clone https://github.com/se7enxweb/exp_static_html_imports.git extension/exp_static_html_imports
Activate it in settings/override/site.ini.append.php:
[ExtensionSettings] ActiveExtensions[]=exp_static_html_imports
Deploy, check, look first, then import:
php bin/php/ezpgenerateautoloads.php -e php bin/php/ezcache.php --clear-all ./console exp:static-import:types ./console exp:static-import:run --markdown --source=/path/to/docs --parent-node=2 --dry-run ./console exp:static-import:run --markdown --source=/path/to/docs --parent-node=2 ./console exp:static-import:status
- exp:static-import:types lists the four types: --doc-html, --html-generic, --markdown, --sitemap-crawl.
- --dry-run counts what would be created, updated, flagged and removed, writes a report, and changes nothing.
- Each Markdown file (.md, .markdown) becomes a Folder below node 2: name from the title, short description from the body. README.md and index.md are the index page of their folder. Front matter is read; code blocks keep their language.
- On Velocity, ./console exp:velocity deploy --allow-root-user replaces the two deploy lines.
Example 2: a folder of HTML, or a live website
./console exp:static-import:run --html-generic --source=/path/to/html --parent-node=2 --dry-run ./console exp:static-import:run --html-generic --source=/path/to/html --parent-node=2 ./console exp:static-import:run --sitemap-crawl --source=https://www.example.com --parent-node=2
- --html-generic reads .html and .htm files; index.html is the folder page. The title comes from h1, the body from main, article or body; nav, header and footer are removed; code in pre keeps its language. Linked .pdf and .zip files become File objects, other assets Image objects.
- --sitemap-crawl reads the pages in the site's sitemap.xml politely: robots.txt respected, one request a second, its own user agent, and each page fetched once.
- --path='docs/*' imports only the matching paths (shell patterns) and --limit=50 only 50 pages: a quick first look at a large source.
Your own classes and sources
Copy the type's group from extension/exp_static_html_imports/settings/statichtmlimport.ini.append.php into settings/override/statichtmlimport.ini.append.php and change it. For example, HTML pages as Articles:
[Type_html-generic] SourceLocation=/path/to/html ParentNode=2 DefaultClass=article AttributeMap[] AttributeMap[article/title]=title AttributeMap[article/body]=body
- Source: SourceLocation (a folder) or SourceRepository (a git URL the import clones and pulls).
- Place: ParentNode, or ParentRemoteID for a node with that remote id.
- Classes: DefaultClass, ClassRules[] (a class per path pattern) and AttributeMap[] (class/attribute=field).
- Rules are INI by default, or YAML, or a PHP rule class (RulesFormat).
- A source type of your own is a PHP class implementing expStaticImportTypeInterface, registered in statichtmlimport.ini [ImportTypes] TypeList[].
- Every rule is described in the extension's doc/rules.md, also shown in the help center when the extension is active.
Example 3: your own copy of the doc.ez.no archive
git clone https://github.com/se7enxweb/exp_static_html_imports.git extension/exp_static_html_imports git clone https://github.com/se7enxweb/exp_doc.git extension/exp_doc
Activate both in settings/override/site.ini.append.php, in this order:
[ExtensionSettings] ActiveExtensions[]=exp_static_html_imports ActiveExtensions[]=exp_doc
Give the folder that will be the documentation site's root the remote id exp_doc_site_root, then:
php bin/php/ezpgenerateautoloads.php -e php bin/php/ezcache.php --clear-all ./console exp:migration:run --extension=exp_doc ./console exp:doc:import --path='FAQ/*' --dry-run ./console exp:doc:import --path='FAQ/*' ./console exp:doc:import
- The migrations create the class group Documentation with its classes (pages, sections, manuals, versions, tutorials and their steps, FAQ groups and their questions, extension reference pages, images and files, each with a Bootstrap icon and a metadata attribute), the sections Doc, Media and Hidden, the role Doc reader for anonymous read access, and the media folder.
- exp:doc:import clones the archive from its repository by itself and imports it with the documentation's rules (exp_doc/settings/statichtmlimport.ini.append.php). It takes the same options as exp:static-import:run.
- --path='FAQ/*' imports one part in minutes. The whole archive is 88,680 pages: its final run took 69 minutes on our server. An interrupted run continues where it stopped when you run the same command again.
- Design: set SiteDesign=doc (Modern) or SiteDesign=doc2025 (Classic) in the front siteaccess's site.ini.append.php. The header switches between them with one click.
- Cronjobs: run runcronjobs.php frequent every five minutes. It reads the front page's commit and forum feeds and recounts the Contributors page; without it both still work, more slowly.
- SQLite: after a large import run PRAGMA analysis_limit = 1000; ANALYZE; again (a migration does it once). It changes no data and keeps content fetches fast.
- Duplicate contributors: ./console exp:doc:contributor-merge --into=<contributor> --from=<duplicate> [--dry-run].
One run is enough; running again is safe
- Identity: every page is the object with remote id staticimport-<type>-<md5 of its source path>, wherever it has been moved.
- Changed source: a new version is published. Unchanged: nothing is written.
- Edited in the CMS: the page is flagged and left alone; --overwrite-flagged writes it anyway and the edit is lost.
- Gone from the source: the node is hidden and reported; --apply-removals moves it to the trash.
- Copies of a page and redirect stubs become redirects to it; error pages are skipped. Every old path, with and without .html, redirects to its new node.
- Large sources run in batches (--batch, 50 by default) and save their position after each; the same command resumes, --restart starts over, --no-pull skips the git pull.
- Every run writes a JSON report (schema exp-static-import/1), a Markdown summary and audit trail events.
- In the administration, Setup > Static site import shows the runs, starts one as a background job, and opens each report with one click.
Questions, ideas and pull requests are welcome on GitHub. Enjoy!
Hello Inkernet!
I apologize for the lack of updates here in The Exponential Blog (and social media in general).
7x has at the direction of myself and the foundation been working harder and longer than ever before to build the next release of Exponential (6.0.15stable). I have not done much but around the clock development to solve even more problems facing users today out of the box in a new installation of Exponential preventing users from needing to pay for the features or code or general knowledge to power their website in today's modern era.
In short be forewarned 6.0.15stable will change the face of the internet; not all at once of course :)
The people who respect never fail operations and features provided before they need to pay for additional development just to make use of the products from a vendor will adopt solutions that scale in advance of needs with zero to marginal cost over non free offerings.
"I get a steady stream on non-free offers from lost souls who have no value to offer; I smirk because we are giving away our best and brightest for zero to marginal cost and these other internet spammers just can't compete with FREEDOM."
Also in the news recently (after a lot of very hard work to free the users!): The Exponential Foundation and 7x Releases The Brand New Exponential Documentation Portal!
We are in the finishing stages of release preparation and you can see the work in advance of the official release at our online demo.
https://demo.exponential.earth
https://edit.demo.exponential.earth
User: admin
Password: publish
We hope you checkout the new documentation which covers the features added which are very very extensive and ground breaking.
https://doc.exponential.earth/exphelp/viewer/changelogs/6.0/6.0.15.md
Be well to each other and take it eZ!
FYI we are seeking new Exponential Partners to support our rapid development pace in the market.
Contact Graham Brookins if you are interested in supporting our team / foundation / projects / products / community financially with a sustaining support subscription and sponsor the exponential project's future.
7x Exponential beats your website stack and we have the proof in the open for everyone to study and learn from.
Cheers,
Graham
CEO @ 7x
Chairman of the Board @ Exponential Foundation
Add comment