4e Compendium (v3.5) Downloader
This is a program for reading D&D Compendium, the official database of Dungeons & Dragons™ 4th edition resources, and save it for offline use. An active (paid) account is required to run the initial download.
How to Use
-
You need an active (paid) D&D Insider subscription.
You can check / recover your DDI account from its official site, and can buy subscription from Digitial River. This program cannot access protected content without an active subscription. -
Fill the login in Data tab.
This downloader relies on you to fill in correct DDI login information. It'll only send your login to the DDI site. Password is remembered on your pc (but not with saved data); delete it after download to be safe. -
Click "Download" and leave it running.
The download process takes at least a day or longer. Progress is saved in real time, so it can be stopped mid-way and it will resume from there next time. The download step requires Internet connection and an active subscription. -
Click "Export" and save it somewhere.
Once data is complete, the download button will become export button. Export is a process of converting downloaded data into a form that web browsers can read. It starts anew every time, usually takes a few minutes, and need 1GB memory. -
(Optional) Transfer the exported file and data to mobile devices.
The exported.html
file can be opened in most browser, and it comes with a same named folder that contains the data. You can copy the file and the folder to other PC or mobile devices. - If you find a typo or mistake in the data, please file an issue. I cannot update the official compendium, but I can update this downloader's corrections.
Troubleshoot (Download)
-
How to get DDI subscription?
- First, you to login or create a wizards account, so that you can login to DDI.
- A new subscriptions can be purchased from Digitial River, which handles subscription payments.
- Once paid, you can disable auto-renew by viewing the new order and follow its manage link.
-
Cannot open / read / write database
Data is stored in "dnd4_compendium.database", and a backup is normally in the same folder which you can try to manually restore.
If that doesn't help, try "Clear Data" in option tab. All data has to be re-downloaded after clear. -
Connection timeout
Increase the timeout period in option tab; long pages like power list will take a while to retrieve. Can also mean a broken or blocked Internet connection. -
Connection reset
Try increase retry count in option tab, or try a more stable network. -
Slow download
Whoever is patient has great understanding, but one who is quick-tempered displays folly. - Proverbs 14:29
You may decrease the throttle rate in option tab, but there are 25k pieces of data to download after all. That is 14 hours if you can crawl at 1 page per 2 seconds. -
Slow export
Temporary disable anti-virus before export. (Windows has a default AV called Windows Defender.) You can also add the export location to the anti-virus' exclusion list.
Alternatively, get a faster computer. My development PC can go from startup to finish export in 8 seconds. -
~ failed (script error)
This means the login form or compendium has changed (again). Usually need a programmer to solve this one. See "other download errors" below. -
Unknown host
1) Network issue. Try again later.
2) Your network is blocking the site, for example some company networks block gaming sites.
3) Wizards finally pulled the plug on ddi.wizards.com. -
Other download errors
Some one-off problems can be solved by closing and rerunning the program.
For persistent problems, if you enable debug tabs and restart, you can see a detailed log, and can also see and control the embedded browser. To report an issue, please include the log in addition to the steps to reproduce.
In case this program is no longer maintained, some errors may be solved by using another Java or an older Java, such as Oracle Java SE 1.8.0.121.
Alternatively, some problems may be solvable with the embedded browser. For example if there is a login problem because Wizards changed the login system, you can try to manually login from this browser. This may allows downloads to proceed normally.
Troubleshoot (Export)
-
Not enough space
Check the drive's free space. Exported data takes up 60 MB, but some devices may require 100+ MB.
You may also opt to compress data, which reduce data size to 12 MB. - Out of Memory This one is always hard. You may get this error, or the program may become unstable. Try hiding debug log, disable data compression, set thread count to 1, and re-launch the program to run a fresh export.
-
Data is wrong / different from official compendium
Known errors of the official compendium will be fixed during export, in addition to other enhancements. See project homepage for a summary, and the in-app debug log (if enabled) will log each affected entry.
Alternatively, you may uncheck "Fix and enhance data" in the Option tab to skip these changes. -
Data is wrong / different from rulebook
Please file an issue and I'll confirm and fix it in next release.
Note that the data need to be re-exported to apply new fixes. -
How to extract the data?
Go to options tab, click on "Dump Data", and select file type from the save file dialog. -
I want the data in XXX format.
You can file an issue to request it.
The current format list is pretty comprehensive and representative of how far I'd go.
No txt because txt has no standard. No Oracle SQL because it is too different. No Word or pdf because it's almost impossible. etc. -
Dumped data is incomplete
1) Official data contains empty entries, and some other miss a good chunk of their content.
2) The tools that read the dump have its limits.
For example Excel has a 32k character limits per cell. It will chop off longer data when it reads the csv/tsv dump.
3) The raw (unfixed) data depends on official compendium's online images and css.
If you dump the data in HTML format, the original view will be preserved. -
Cannot import dumped data
1) See above question, especially the limit part.
2) The data contains Unicode characters and are in UTF-8 format.
Program that does not use Unicode by default may display garbage or throw error.
3) For SQL, the export is tested on these databases at default settings:
MySQL 5.7, SQL Server 2016, and Postgre 9.6.
Older versions may not support the syntax being used; usually they are not hard to fix yourself.
4) For Excel (xlsx), the export opens without error in Excel 2013.
As of Jan 2017, Google Sheet cannot import the data regardless of the format for unknown reason. If you ask for their support and they reply something useful, please file an issue to let me know.
Troubleshoot (HTML)
-
"Please Enable JavaScript"
1) "Internet Explorer" (Windows): When the page is loaded, a bar would popup at the bottom that allow you to enable JavaScript. The bar will disappear after a while, but you can reload the page to get it back.
2) "HTML Viewer" (Android): Choice a proper browser, such as Chrome. HTML Viewer is not a browser.
3) No browser choice on Android: Clear default browser (Settings > Apps > Reset default apps) first. If there is still no choice, some apps will use their own internal browser when you click on an HTML file. Use a proper file manager instead (see next point). -
"Cannot load content script"
Some file managers on Android will wrap the file into a "content://" url. We can't load data from a content url because we don't know the real path.
A real file manager such as ES File Explorer or Total Commander will use the correct "file://" url. -
"Script error"
This should not happen if you used a release version to export the data. First thing to try should be to upgrade your browser. See Homepage for a list of tested browsers. -
"Cannot read XX from xxx/yyy/zzz.js"
This happens when the data files are corrupted. If clear cache does not help, try re-export the data. Note this this is not caused by download data corruption, please don't re-download data. -
Blank / "null" / "undefined" list or content
This is usually caused by coding error and should be reported with detailed steps to reproduce.
About
This downloader and the exported compendium is open source, licensed under GNU Affero GPL v3. Complete source code is available from project homepage on github.
This downloader uses SQLJet (v1.1), licensed under GNU GPL v2. SQLJet uses ANTLR (v3.5), licensed under 3-Clause BSD. Program icon derived from RRZEicons's database and pixabay's dragon.
The free licenses apply to code and related resources, not to the data you downloaded.
Dungeons & Dragons is a trademark of Wizards of the Coast LLC.