Difference between revisions of "Static Data Export"

From EVE University Wiki
Jump to: navigation, search
(Added a brief guide about using the SDE, changed it to the new yaml formats. The table still needs some work.)
Line 1: Line 1:
{{update|It's no longer an MSSQL Format, it's provided by CCP only as .static and .yaml files. MSSQL still available from FuzzySteve's conversions. Also some ongoing changes to the SDE's format (fsd/bsd)}}
+
CCP provides developers a series of static files, known as the Static Data Export(SDE), which contains static data from the Tranquility server. The SDE is currently exported as .yaml and .static files, however, different and easier to use formats(mentioned below) are made by the community. The SDE can be found at the EVE developers [https://developers.eveonline.com/resource/resources resource] page. All resources provided by CCP are subject to the [https://developers.eveonline.com/resource/license-agreement license agreement] linked on page.
 
 
CCP makes available a Microsoft SQL Server data dump file of static data, as it appears in Tranquility Server, for use by the EVE community. This data dump is called the Static Data Export. Along with this dump Image Export collections are also made available. These resources can be found at the EVE developers [https://developers.eveonline.com/resource/resources resource] page. All resources provided by CCP are subject to the [https://developers.eveonline.com/resource/license-agreement license agreement] linked on page.
 
  
 
= Fuzzwork SDE Conversions =
 
= Fuzzwork SDE Conversions =
To aide fellow developers and players in consuming this data without having to extract or convert from Microsoft SQL Server format every time Steve Ronuken has hosted conversions in PostgreSQL, SQLite and MySQL readable formats. They can be found at [https://www.fuzzwork.co.uk Fuzzwork] under SDE or more directly at [https://www.fuzzwork.co.uk/dump www.fuzzwork.co.uk/dump]. Individual table data can be found in [//en.wikipedia.org/wiki/Comma-separated_values CSV] and [//en.wikipedia.org/wiki/SQL SQL] (MySQL) formats here [https://www.fuzzwork.co.uk/dump/latest www.fuzzwork.co.uk/dump/latest].
+
To aide fellow developers and players in consuming this data without having to extract or convert from YAML or static format every time Steve Ronuken has hosted conversions in PostgreSQL, SQLite, MySQL, MSSQL, and CSV formats. They can be found at [https://www.fuzzwork.co.uk Fuzzwork] under SDE or more directly at [https://www.fuzzwork.co.uk/dump www.fuzzwork.co.uk/dump]. Individual table data can be found in [//en.wikipedia.org/wiki/Comma-separated_values CSV] and [//en.wikipedia.org/wiki/SQL SQL] (MySQL) formats here [https://www.fuzzwork.co.uk/dump/latest www.fuzzwork.co.uk/dump/latest].
 
== Opening a file from Fuzzwork SDE Conversions ==
 
== Opening a file from Fuzzwork SDE Conversions ==
To extract data from compressed files, use [https://ss64.com/bash/tar.html tar] for <code>.tar.bz2</code> files or [https://ss64.com/osx/bzip.html bunzip2] for <code>.bz2</code> files in linux, and something like [http://www.7-zip.org/ 7-zip] for either format in windows.
+
To extract data from compressed files, use [https://ss64.com/bash/tar.html tar] for <code>.tar.bz2</code> files or [https://ss64.com/osx/bzip.html bunzip2] for <code>.bz2</code> files in linux, and something like [http://www.7-zip.org/ 7-zip] for either format in windows. Once extracted the files can be opened in the appropriate application that supports the file format.
Once extracted the files can be opened in the appropriate application that supports the file format.
+
 
 +
= How does the ESI fit into all this? =
 +
The [[Eve Swagger Interface]] intends to have endpoints to account for all of the SDE, currently, it's not quite there yet, and the SDE is needed for a lot a projects. The progress can be tracked [https://github.com/esi/esi-issues/issues/1103 here].
 +
 
 +
= How to implement the original SDE into your application =
 +
 
 +
== Step 0: Preface ==
 +
This is not a guide to Python or programming, if you don't know how to program, there are tons of resources to learn to code, there are a few of my favorites in the External Links section down below, if your having issues, contact me at [[User:Asgore Astroficus|Asgore Astroficus]] ([[User talk:Asgore Astroficus|talk]]).
 +
 
 +
== Step 1: Opening the file ==
 +
Firstly, find the file containing the data you need, for this example, I'm going to use fsd/blueprints.yaml. This file contains all the information required to build an industry cost predictor. Next, open the file using your desired programming language, I'm going to use python, so to open this I'll run ''with open("fsd/blueprints.yaml", "r") as f:''. The file is open! To the next step!
 +
 
 +
== Step 2: Reading the file ==
 +
=== Step 2.1: Installing PyYAML ===
 +
Assuming your using pip, this should be as simple as ''pip install pyyaml''. If you having issues, contact me at [[User:Asgore Astroficus|Asgore Astroficus]] ([[User talk:Asgore Astroficus|talk]]) or have a look at [https://stackabuse.com/reading-and-writing-yaml-to-a-file-in-python/ this article].
 +
 
 +
=== Step 2.2: Interpreting the file ===
 +
Now that PyYAML is installed, we need to use it to parse our file, we have already opened it, so it should just be a matter of parsing it with PyYAML, which can be done by ''blueprints = yaml.load(f, Loader=yaml.FullLoader)'', remember to have ''import yaml'' at the top of your .py file. Hopefully now if you print out the blueprints variable, your console will be filled with sweet, sweet data and the taste of success. The data will be a dictionary, and the rest from here should be relatively simple, it should just be navigating through the data as if it was a dictionary.
 +
 
 +
== Step 3: Postface ==
 +
I hope this short guide helped you somewhat getting into the SDE, I might write one for the Fuzzwork conversions soon. If you have any issues, pop me a message here [[User:Asgore Astroficus|Asgore Astroficus]] ([[User talk:Asgore Astroficus|talk]]).
  
== What each table contains ==
+
= What each table contains =
 
{| class="wikitable sortable"
 
{| class="wikitable sortable"
 
|-
 
|-
Line 191: Line 209:
 
|}
 
|}
  
== External links ==
 
* [https://eveonline-third-party-documentation.readthedocs.io/en/latest/sde/index.html Documentation of the Static Data Export]
 
  
 
[[Category:Applications]]
 
[[Category:Applications]]

Revision as of 19:49, 11 June 2020

CCP provides developers a series of static files, known as the Static Data Export(SDE), which contains static data from the Tranquility server. The SDE is currently exported as .yaml and .static files, however, different and easier to use formats(mentioned below) are made by the community. The SDE can be found at the EVE developers resource page. All resources provided by CCP are subject to the license agreement linked on page.

Fuzzwork SDE Conversions

To aide fellow developers and players in consuming this data without having to extract or convert from YAML or static format every time Steve Ronuken has hosted conversions in PostgreSQL, SQLite, MySQL, MSSQL, and CSV formats. They can be found at Fuzzwork under SDE or more directly at www.fuzzwork.co.uk/dump. Individual table data can be found in CSV and SQL (MySQL) formats here www.fuzzwork.co.uk/dump/latest.

Opening a file from Fuzzwork SDE Conversions

To extract data from compressed files, use tar for .tar.bz2 files or bunzip2 for .bz2 files in linux, and something like 7-zip for either format in windows. Once extracted the files can be opened in the appropriate application that supports the file format.

How does the ESI fit into all this?

The Eve Swagger Interface intends to have endpoints to account for all of the SDE, currently, it's not quite there yet, and the SDE is needed for a lot a projects. The progress can be tracked here.

How to implement the original SDE into your application

Step 0: Preface

This is not a guide to Python or programming, if you don't know how to program, there are tons of resources to learn to code, there are a few of my favorites in the External Links section down below, if your having issues, contact me at Asgore Astroficus (talk).

Step 1: Opening the file

Firstly, find the file containing the data you need, for this example, I'm going to use fsd/blueprints.yaml. This file contains all the information required to build an industry cost predictor. Next, open the file using your desired programming language, I'm going to use python, so to open this I'll run with open("fsd/blueprints.yaml", "r") as f:. The file is open! To the next step!

Step 2: Reading the file

Step 2.1: Installing PyYAML

Assuming your using pip, this should be as simple as pip install pyyaml. If you having issues, contact me at Asgore Astroficus (talk) or have a look at this article.

Step 2.2: Interpreting the file

Now that PyYAML is installed, we need to use it to parse our file, we have already opened it, so it should just be a matter of parsing it with PyYAML, which can be done by blueprints = yaml.load(f, Loader=yaml.FullLoader), remember to have import yaml at the top of your .py file. Hopefully now if you print out the blueprints variable, your console will be filled with sweet, sweet data and the taste of success. The data will be a dictionary, and the rest from here should be relatively simple, it should just be navigating through the data as if it was a dictionary.

Step 3: Postface

I hope this short guide helped you somewhat getting into the SDE, I might write one for the Fuzzwork conversions soon. If you have any issues, pop me a message here Asgore Astroficus (talk).

What each table contains

Table name What it contains
agtAgents.csv.bz2 Contains information regarding agents such as location, corporation, type, level, and locator (boolean)
agtAgentTypes.csv.bz2 Example
agtResearchAgents.csv.bz2 Example
certCerts.csv.bz2 Example
certMasteries.csv.bz2 What ships have what certificates. In this case, typeID is the ship, and certID is the certificate. See certSkills
certSkills.csv.bz2 Example
chrAncestries.csv.bz2 Example
chrAttributes.csv.bz2 Example
chrBloodlines.csv.bz2 Contains information on what race each bloodline belongs to, descriptions.
chrFactions.csv.bz2 Faction IDs, name, and description. Also what race each faction belongs to, and some other misc. information.
chrRaces.csv.bz2 raceID and raceName, iconID and a long and short description (side note, ORE is a race)
crpActivities.csv.bz2 IDK, maybe names of NPC sturcture groups?
crpNPCCorporationDivisions.csv.bz2 Example
crpNPCCorporationResearchFields.csv.bz2 Unknown
crpNPCCorporations.csv.bz2 Example
crpNPCCorporationTrades.csv.bz2 Information on what each NPC corp sells.
crpNPCDivisions.csv.bz2 Information on NPC divisions, mostly deprecated.
dgmAttributeCategories.csv.bz2 What each categoryID is, and a categoryName
dgmAttributeTypes.csv.bz2 Example
dgmEffects.csv.bz2 Information regarding effects, such as launching missiles, scaling damage, and the effects skills have.
dgmExpressions.csv.bz2 Information regarding expressions
dgmTypeAttributes.csv.bz2 AttributeID of each item (typeID). See dgmAttributeTypes
dgmTypeEffects.csv.bz2 effectID that each typID had, see also dgmEffects.csv.bz2
eveGraphics.csv.bz2 Information regarding grapics such as graphicID, graphic location, race, and description.
eveIcons.csv.bz2 Example
eveUnits.csv.bz2 Example
industryActivity.csv.bz2 Contains the amount of seconds it takes to create a blueprint product
industryActivityMaterials.csv.bz2 Contains the materials required to create a blueprint product
industryActivityProbabilities.csv.bz2 Contains the outcome and the probability of an invention
industryActivityProducts.csv.bz2 Shows what processes create what (such as what products a blueprint creates)
industryActivityRaces.csv.bz2 Example
industryActivitySkills.csv.bz2 Contains information about the skills required to manufacture any blueprint.
industryBlueprints.csv.bz2 Contains the base maximum amount of BPC runs
invCategories.csv.bz2 Category IDs and category names
invContrabandTypes.csv.bz2 Example
invControlTowerResourcePurposes.csv.bz2 Example
invControlTowerResources.csv.bz2 Example
invFlags.csv.bz2 Example
invGroups.csv.bz2 Contains information on inventory groups, such as groupID, categoryID, and other metadata
invItems.csv.bz2 Information about itemID and typeID, and some others
invMarketGroups.csv.bz2 Example
invMetaGroups.csv.bz2 A small file about meta group ids, such as Tech I, Tech II, Faction, Officer, etc
invMetaTypes.csv.bz2 typeID, parentTypeID, metaGroupID
invNames.csv.bz2 itemID for everything. Not that itemID is not typeID.
invPositions.csv.bz2 Example
invTraits.csv.bz2 Example
invTypeMaterials.csv.bz2 Example
invTypeReactions.csv.bz2 Example
invTypes.csv.bz2 Example
invUniqueNames.csv.bz2 Example
invVolumes.csv.bz2 Example
mapCelestialStatistics.csv.bz2 Example
mapConstellationJumps.csv.bz2 Example
mapConstellations.csv.bz2 Constellation position, faction, name, region, name, and radius.
mapDenormalize.csv.bz2 Position, radius, and security status of moons, planets, solar systems, all the way to regions. As such, a very large CSV with more than half a million lines.
mapJumps.csv.bz2 Example
mapLandmarks.csv.bz2 Example
mapLocationScenes.csv.bz2 Example
mapLocationWormholeClasses.csv.bz2 Each location's wormholeClassID
mapRegionJumps.csv.bz2 Example
mapRegions.csv.bz2 Example
mapSolarSystemJumps.csv.bz2 Example
mapSolarSystems.csv.bz2 Example
mapUniverse.csv.bz2 Possibly to differentiate between K-space and J-space? (speculation)
planetSchematics.csv.bz2 schematicID, name, and time it takes to complete that schematic.
planetSchematicsPinMap.csv.bz2 Example
planetSchematicsTypeMap.csv.bz2 Example
ramActivities.csv.bz2 Describes what each R.A.M. does. See Manufacturing
ramAssemblyLineStations.csv.bz2 Example
ramAssemblyLineTypeDetailPerCategory.csv.bz2 Information on R.A.M.
ramAssemblyLineTypeDetailPerGroup.csv.bz2 Example
ramAssemblyLineTypes.csv.bz2 Example
ramInstallationTypeContents.csv.bz2 Example
skinLicense.csv.bz2 Example
skinMaterials.csv.bz2 Example
skins.csv.bz2 Example
skinShip.csv.bz2 What skins can be used on what ships
staOperations.csv.bz2 Example
staOperationServices.csv.bz2 Information about what station types offer what services (speculation)
staServices.csv.bz2 Example
staStations.csv.bz2 Information about stations, their security level, location, name, reprocessing level, and much more.
staStationTypes.csv.bz2 Example
translationTables.csv.bz2 Appears to be empty
trnTranslationColumns.csv.bz2 Example
trnTranslationLanguages.csv.bz2 Example
trnTranslations.csv.bz2 Example
warCombatZones.csv.bz2 A small file that updates regularly. Contains information about combat zones (FW I think?)
warCombatZoneSystems.csv.bz2 Example