Build definitions from local repository?

Question

Build definitions from local repository?

Closed this issue 5 years ago · 9 comments

Hi everyone. I have several OVAL files with the same objects and states which i use in OVALdi. I used oval_decomposition.py to make it more manageable, but i can not build definition back. I made empty git repository and one test commit and then i get this:

File "C:\Program Files (x86)\Python37-32\lib\site-packages\whoosh\fields.py", line 840, in to_bytes
x = self.prepare_datetime(x)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\whoosh\fields.py", line 819, in prepare_datetime
x = self._parse_datestring(x)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\whoosh\fields.py", line 872, in _parse_datestring
raise Exception("%r is not a parseable date" % qstring)
Exception: '' is not a parseable date

I assume it is because i do not have full OVALRepo, but i really do not need it. Is there a good way to build xml files from decomposition locally?

Answer 1 · 2019-08-09T13:16:35.000Z

Hi @DenisNovac,

You're right that these tools were built and tested to support maintenance of the entire oval repository... but... I don't know why they wouldn't work on your own custom, mini-repository.

The error you're getting indicates that the content is missing a datetime value somewhere that the repository requires a datetime to be. We should probably improve the error message to give some more context.

Can you provide the entire stack trace?

-David

Answer 2 · 2019-08-09T13:29:29.000Z

@DavidRies, sure. I'll start from beginning.
Here i run decomposition and then try to rebuild it back:

PS C:\Users\dyablochkin\Documents\dev\oval\scripts> python .\oval_decomposition.py -f .\wincc_cve_2019_10935.xml
PS C:\Users\dyablochkin\Documents\dev\oval\scripts> python .\build_oval_definitions_file.py -o result.xml --family windows
Traceback (most recent call last):
  File ".\build_oval_definitions_file.py", line 181, in <module>
    main()
  File ".\build_oval_definitions_file.py", line 105, in main
    query_results = definitions_index.query(query) if not all_definitions_filtered else {}
  File "C:\Users\dyablochkin\Documents\dev\oval\scripts\lib_search.py", line 91, in query
    self.update()
  File "C:\Users\dyablochkin\Documents\dev\oval\scripts\lib_search.py", line 190, in update
    if not self.index_based_on_current_commit():
  File "C:\Users\dyablochkin\Documents\dev\oval\scripts\lib_search.py", line 267, in index_based_on_current_commit
    current_commit = lib_git.get_current_commit_hash()
  File "C:\Users\dyablochkin\Documents\dev\oval\scripts\lib_git.py", line 105, in get_current_commit_hash
    repo = get_repo()
  File "C:\Users\dyablochkin\Documents\dev\oval\scripts\lib_git.py", line 110, in get_repo
    return git.Repo(lib_repo.get_root_path())
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\git\repo\base.py", line 183, in __init__
    raise InvalidGitRepositoryError(epath)
git.exc.InvalidGitRepositoryError: C:\Users\dyablochkin\Documents\dev\oval

Then i try to initialize git repo and get another error:

PS C:\Users\dyablochkin\Documents\dev\oval\scripts> cd ..
PS C:\Users\dyablochkin\Documents\dev\oval> git init
Initialized empty Git repository in C:/Users/dyablochkin/Documents/dev/oval/.git/
PS C:\Users\dyablochkin\Documents\dev\oval> cd .\scripts\
PS C:\Users\dyablochkin\Documents\dev\oval\scripts> python .\build_oval_definitions_file.py -o result.xml --family windows
Traceback (most recent call last):
  File ".\build_oval_definitions_file.py", line 181, in <module>
    main()
  File ".\build_oval_definitions_file.py", line 105, in main
    query_results = definitions_index.query(query) if not all_definitions_filtered else {}
  File "C:\Users\dyablochkin\Documents\dev\oval\scripts\lib_search.py", line 91, in query
    self.update()
  File "C:\Users\dyablochkin\Documents\dev\oval\scripts\lib_search.py", line 190, in update
    if not self.index_based_on_current_commit():
  File "C:\Users\dyablochkin\Documents\dev\oval\scripts\lib_search.py", line 267, in index_based_on_current_commit
    current_commit = lib_git.get_current_commit_hash()
  File "C:\Users\dyablochkin\Documents\dev\oval\scripts\lib_git.py", line 106, in get_current_commit_hash
    return str(repo.head.commit)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\git\refs\symbolic.py", line 200, in _get_commit
    obj = self._get_object()
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\git\refs\symbolic.py", line 193, in _get_object
    return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\git\refs\symbolic.py", line 135, in dereference_recursive
    hexsha, ref_path = cls._get_ref_info(repo, ref_path)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\git\refs\symbolic.py", line 184, in _get_ref_info
    return cls._get_ref_info_helper(repo, ref_path)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\git\refs\symbolic.py", line 167, in _get_ref_info_helper
    raise ValueError("Reference at %r does not exist" % ref_path)
ValueError: Reference at 'refs/heads/master' does not exist

Then i make one commit and here we go:

PS C:\Users\dyablochkin\Documents\dev\oval\scripts> cd ..
PS C:\Users\dyablochkin\Documents\dev\oval> git add .\test.txt
PS C:\Users\dyablochkin\Documents\dev\oval> git commit -m "test"
[master (root-commit) 8a854a9] test
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 test.txt

PS C:\Users\dyablochkin\Documents\dev\oval> cd .\scripts\
PS C:\Users\dyablochkin\Documents\dev\oval\scripts> python .\build_oval_definitions_file.py -o result.xml --family windows
INFO: Rebuilding oval_definitions index completed (1 definitions)
Process SubWriterTask-1:
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\whoosh\multiproc.py", line 113, in run
    self._process_file(*jobinfo)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\whoosh\multiproc.py", line 149, in _process_file
    writer.add_document(**args)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\whoosh\writing.py", line 750, in add_document
    for tbytes, freq, weight, vbytes in items:
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\whoosh\fields.py", line 669, in index
    yield (self.to_bytes(num, shift), 1, 1.0, emptybytes)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\whoosh\fields.py", line 840, in to_bytes
    x = self.prepare_datetime(x)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\whoosh\fields.py", line 819, in prepare_datetime
    x = self._parse_datestring(x)
  File "C:\Program Files (x86)\Python37-32\lib\site-packages\whoosh\fields.py", line 872, in _parse_datestring
    raise Exception("%r is not a parseable date" % qstring)
Exception: '' is not a parseable date
INFO: Found 0 matching OVAL definitions
INFO: Writing OVAL definitions to result.xml
INFO: Completed in 00:00:01!

Actually yes, First time this script gives me an xml, but it is empty default-looking file:

<?xml version="1.0" encoding="UTF-8"?>
<oval_definitions
	xmlns="http://oval.mitre.org/XMLSchema/oval-definitions-5"
	xmlns:oval="http://oval.mitre.org/XMLSchema/oval-common-5"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://oval.mitre.org/XMLSchema/oval-common-5 oval-common-schema.xsd http://oval.mitre.org/XMLSchema/oval-definitions-5 oval-definitions-schema.xsd">

	<generator>
		<oval:product_name>CIS OVAL Repository</oval:product_name>
		<oval:product_version>0.1</oval:product_version>
		<oval:schema_version>5.11.2</oval:schema_version>
		<oval:timestamp>2019-08-09T13:24:30</oval:timestamp>
	</generator>
		
</oval_definitions>

On the next run there is no output at all.

Answer 3 · 2019-08-09T13:59:12.000Z

Hi @DenisNovac, thanks for the additional context. You are feeling the pain of using these scripts in a different context than they were intended for. Here's what I think is going on:

git.exc.InvalidGitRepositoryError: these scripts must be run in a valid git repo. You've figured that out!
Reference at 'refs/heads/master' does not exist: there must be at least one commit. The whoosh-based searching uses the commit hash when determining whether or not it needs to rebuild the index. Again, you've solved this by making a commit.

That gets us to Exception: '' is not a parseable date. I expect that one or more the OVAL definitions in that you decomposed does not have any elements under metadata/oval_repository/dates/.

Repository definitions are required to have entries there (see https://github.com/CISecurity/OVALRepo/blob/master/CONTRIBUTING.md):

Within the <oval_repository>/ element, the new definition MUST include a element indicating the date of submission, as well as the sub-element indicating the organization and user submitting the definition.

The scripts expect at least one date in each definition and use that to determine the last modified date of the definition.

Let me know if that solves your issue.

Answer 4 · 2019-08-10T19:24:12.000Z

Hi @DavidRies. I tried to add this block in my files definition:

<oval_repository>
                    <dates>
                        <submitted date="2019-08-11T01:00:00.000+05:00">
                            <contributor organization="USSC">Denis Yablochkin</contributor>
                        </submitted>
                    </dates>
</oval_repository>

And my build was successful:

PS C:\Users\denis\Documents\dev\OVALRepo\scripts> python .\build_oval_definitions_file.py --family windows -o result.xml
INFO: Rebuilding oval_definitions index completed (1 definitions)
INFO: Found 1 matching OVAL definitions
INFO: Finding downstream OVAL ids for all definitions
INFO: Rebuilding oval_elements index completed (15 elements)
INFO: Found 14 downstream OVAL ids
INFO: Finding paths for 15 OVAL elements
INFO: Generating OVAL definition file with 15 elements
INFO: Writing OVAL definitions to result.xml
INFO: Completed in 00:00:02!

But if i tries to do this again i get this:

INFO: Updating oval_definitions index |-    > 1 definitionsTraceback (most recent call last):
  File ".\build_oval_definitions_file.py", line 181, in <module>
    main()
  File ".\build_oval_definitions_file.py", line 105, in main
    query_results = definitions_index.query(query) if not all_definitions_filtered else {}
  File "C:\Users\denis\Documents\dev\OVALRepo\scripts\lib_search.py", line 91, in query
    self.update()
  File "C:\Users\denis\Documents\dev\OVALRepo\scripts\lib_search.py", line 247, in update
    index_writer.add_document(**document)
  File "C:\Users\denis\AppData\Roaming\Python\Python37\site-packages\whoosh\writing.py", line 750, in add_document
    for tbytes, freq, weight, vbytes in items:
  File "C:\Users\denis\AppData\Roaming\Python\Python37\site-packages\whoosh\fields.py", line 669, in index
    yield (self.to_bytes(num, shift), 1, 1.0, emptybytes)
  File "C:\Users\denis\AppData\Roaming\Python\Python37\site-packages\whoosh\fields.py", line 840, in to_bytes
    x = self.prepare_datetime(x)
  File "C:\Users\denis\AppData\Roaming\Python\Python37\site-packages\whoosh\fields.py", line 819, in prepare_datetime
    x = self._parse_datestring(x)
  File "C:\Users\denis\AppData\Roaming\Python\Python37\site-packages\whoosh\fields.py", line 872, in _parse_datestring
    raise Exception("%r is not a parseable date" % qstring)
Exception: '' is not a parseable date

Looks like i have to make new commit before launching the build. Also it works if i delete .git directory and make git init and git commit once more (build from existing repo works too, without re-decomposition in both situations). I can write a workaround for it by myself, so thank you very much :)

Answer 5 · 2019-08-12T15:40:20.000Z

Thanks for the update. Glad to hear you've got it working. I don't understand why it breaks if you rerun it. Do you mean that if you run python .\build_oval_definitions_file.py --family windows -o result.xml and it works and then you immediately run the same command again (without changing any files or running any other commands), it fails the second time?

Answer 6 · 2019-08-12T18:00:19.000Z

@DavidRies, yes. I decompose one file. Then i make git repo and initial commit. Then i build this one file and it all works. If i try to build it once more (even with different search option) - i get exception "not parseable date". If i decompose one more file with other CVE and try to build only this one (with --reference_id, not --family this time), i get the same error.

At beginning i thought it is ok to create new commits with new files. But sometimes i need to build several configs from the same repo (for example, for different CVEs). So for workaround i just delete and recreate git repo and commit every time i need to build something (like this). But i don't know yet if i break something important with it...

Answer 7 · 2019-08-12T18:30:04.000Z

Have you added all of your OVAL files (the ones created using decompose) to your git repo and committed them? I'd recommend this workflow:

create git repo (once)
decompose file into repo
add all .xml files to git and commit all changes
build your .xml
go back to 2 for more/different files

-David

Answer 8 · 2019-08-13T06:29:20.000Z

@DavidRies
Yes, it will work. But sometimes i need to build more than one .xml. Like, for different CVEs. In this situation i do not want to decompose new xmls, make commits, etc. So if i want to build file - i automatically create new git repo at folder above and make commit with empty file. After building i delete it. Seems like xml-files search in repository still works. And decomposition works even without git repository. But maybe there is some git-integration-thing that will break at some point if i continue using library like this (like when my local repository become bigger)?

Answer 9 · 2019-08-13T14:11:32.000Z

I see. These scripts are designed to operate on a persistent git repo with a changing set of content files. It sounds like you've got it working your way, which I'm glad to hear.