fbtest Reference¶
Globals¶
-
fbtest.
DB_NEW
= 'New'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
DB_EXISTING
= 'Existing'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
DB_RESTORE
= 'Restore'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
DB_ACCESS
= [None, 'New', 'Existing', 'Restore']¶ list() -> new empty list list(iterable) -> new list initialized from iterable’s items
-
fbtest.
CHARACTER_SETS
= [None, 'NONE', 'ASCII', 'BIG_5', 'CYRL', 'DOS437', 'DOS737', 'DOS775', 'DOS850', 'DOS852', 'DOS857', 'DOS858', 'DOS860', 'DOS861', 'DOS862', 'DOS863', 'DOS864', 'DOS865', 'DOS866', 'DOS869', 'EUCJ_0208', 'GBK', 'GB_2312', 'ISO8859_1', 'ISO8859_2', 'ISO8859_3', 'ISO8859_4', 'ISO8859_5', 'ISO8859_6', 'ISO8859_7', 'ISO8859_8', 'ISO8859_9', 'ISO8859_13', 'KOI8R', 'KOI8U', 'KSC_5601', 'NEXT', 'OCTETS', 'SJIS_0208', 'TIS620', 'UNICODE_FSS', 'UTF8', 'WIN1250', 'WIN1251', 'WIN1252', 'WIN1253', 'WIN1254', 'WIN1255', 'WIN1256', 'WIN1257', 'WIN1258', 'LATIN2']¶ list() -> new empty list list(iterable) -> new list initialized from iterable’s items
-
fbtest.
PAGE_SIZES
= [None, '1024', '2048', '4096', '8192', '16384', '32768']¶ list() -> new empty list list(iterable) -> new list initialized from iterable’s items
-
fbtest.
TYPE_ISQL
= 'ISQL'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
TYPE_PYTHON
= 'Python'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
TEST_TYPES
= ['ISQL', 'Python']¶ list() -> new empty list list(iterable) -> new list initialized from iterable’s items
-
fbtest.
PLATFORMS
= ['Windows', 'Linux', 'MacOS', 'FreeBSD', 'Solaris', 'HP-UX']¶ list() -> new empty list list(iterable) -> new list initialized from iterable’s items
-
fbtest.
UNKNOWN
= 'Unknown'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
template_base
= '<%def name="title()">Firebird QA Analysis</%def>\n<%! import datetime %>\n\n<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">\n<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8">\n<style type="text/css" media="screen" title="Normal Text">\nhtml, body, div, span, applet, object, iframe, h1, h2, h3, h4, h5, h6, p,\nblockquote, pre, a, abbr, acronym, address, big, cite, code, del, dfn, em,\nfont, img, ins, kbd, q, s, samp, small, strike, strong, sub, sup, tt, var,\ndl, dt, dd, ol, ul, li, fieldset, form, label, legend, table, caption,\ntbody, tfoot, thead, tr, th, td { margin: 0; padding: 0; border: 0; outline: 0;\n font-weight: inherit; font-style: inherit; font-size: 100%; font-family: inherit;\n vertical-align: baseline;}\n:focus { outline: 0; }\nbody { line-height: 1; color: black; background: white; }\nol, ul { list-style: none; }\ncaption, th, td { text-align: left; font-weight: normal; }\nblockquote:before, blockquote:after, q:before, q:after { content: ""; }\nblockquote, q { quotes: "" ""; }\n\nbody { margin: 0; padding: 0; font-family: verdana, sans-serif; font-size: 69%;\n color: #000; background-color: #fff; }\nh1 { font-size: 1.4em; font-weight: bold; margin-top: 0em; margin-bottom: 0em; }\nh2 { font-size: 1.2em; margin: 1.2em 0em 1.2em 0em; font-weight: bold; }\nh3 { font-size: 1.0em; margin: 1.2em 0em 1.2em 0em; font-weight: bold; }\nh4 { font-size: 0.95em; margin: 1.2em 0em 1.2em 0em; font-weight: normal; }\nh5 { font-size: 0.9em; margin: 1.2em 0em 1.2em 0em; font-weight: normal; }\nh6 { font-size: 0.85em; margin: 1.2em 0em 1.2em 0em; font-weight: normal; }\npre { font-family: courier, courier-new; }\n\nth { text-align: center; font-weight: bold; background-color: #e0e0e0; padding: 2px; }\ntable { border-collapse: separate; border-spacing: 2px; }\ntd { padding: 2px 0.5em 0 0.5em;}\n\n.PASS { background-color: #a0f0a0; text-align: }\n.FAIL { background-color: #ff9090; text-align: }\n.ERROR { background-color: #ffffe0; text-align: }\n.UNTESTED { background-color: #d0d0d0; text-align: }\n.SKIPPED { background-color: #d0d0d0; text-align: }\n.NA { background-color: white; text-align: center; }\n.test_id { padding-left: 5px;}\n.ann-o { background-color: white; padding: 2px; }\n.ann-e { background-color: #ffffe0; padding: 2px; }\n.cause { background-color: #ff9090; padding: 2px;}\n.total { font-weight: bold; background-color: #e0e0e0; padding: 2px; }\n\n#Container { margin: 0em auto; padding: 4px 20px 4px 20px ;\n text-align: left; /* Win IE5 */ }\n#Header { position: relative; margin: 0; padding: 10px; clear: both; }\n#Footer { font-size: 0.9em; color: #666; line-height: 1.3em; padding-top: 5px;clear: both; }\n#Content { display: block; }\n\n\n</style>\n<title>${self.title()}</title>\n</head>\n<body>\n\n<div id="Container">\n <div id="Header"><H1>QA Analysis for Firebird ${version}</H1></div>\n\n <div id="Content">${self.body()}\n </div>\n\n <div id="Footer">\n Generated by fbtest analyzer, ${datetime.datetime.now()}\n </div>\n</div>\n\n</body>\n</html>\n'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
template_detail
= '<%inherit file="base.mako"/>\n<H1>Details for test ${test_id}</H1>\n%for group in test_detail:\n<%\nresult = group[0]\ni = 1\nhas_diffs = diffs_only and len([key for key in result.annotations.keys() if key.endswith(\'diff\')]) > 0\nif has_diffs:\n annotations = ((x,y) for x,y in result.annotations.items() if x.endswith(\'diff\'))\nelse:\n annotations = ((x,y) for x,y in result.annotations.items() if x != result.CAUSE)\n%>\n<H2>${\', \'.join(group[1])}</H2>\n<table width="100%" border=1>\n<tr><th>Annotation</th><th>Value</th></tr>\n<tr><td class="cause">Cause</td><td class="cause">${result.get_cause()}</td></tr>\n%for key,value in annotations:\n<tr>\n%if (i % 2) == 0:\n<td class="ann-o">${key}</td><td class="ann-o"><pre>${value}</pre></td>\n%else:\n<td class="ann-e">${key}</td><td class="ann-e"><pre>${value}</pre></td>\n%endif\n</tr>\n<%\ni += 1\n%>\n%endfor\n</table>\n%endfor\n'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
template_main
= '<%inherit file="base.mako"/>\n<%\nfrom operator import attrgetter\nfrom itertools import groupby\n\nplatforms = [(k,len(list(g))) for k,g in groupby(results,attrgetter(\'platform\'))]\ncpuarchs = [(k,len(list(g))) for k,g in groupby(results,attrgetter(\'cpuarch\'))]\narchs = [(k,len(list(g))) for k,g in groupby(results,attrgetter(\'arch\'))]\nruns = [(k,len(list(g))) for k,g in groupby(results,attrgetter(\'sequence\'))]\ntotals = [0 for x in tests[test_order[0]]]\n%>\n<table>\n<tr>\n%for platform,span in platforms:\n <th colspan=${span}>${platform}</th>\n%endfor\n</tr>\n<tr>\n%for cpuarch,span in cpuarchs:\n <th colspan=${span}>${cpuarch}-bit</th>\n%endfor\n</tr>\n<tr>\n%for arch,span in archs:\n <th colspan=${span}>${arch}</th>\n%endfor\n</tr>\n<tr>\n%for run,span in runs:\n <th colspan=${span}>${run}</th>\n%endfor\n</tr>\n%for test_id in test_order:\n<% test_results = tests[test_id] %>\n<!--%for test_id,test_results in tests.items():-->\n<tr>\n%for result in test_results:\n%if result:\n<% r = result.outcome\ntotals[test_results.index(result)] += result.get_elapsed()\n%>\n%else:\n<% r = \'NA\' %>\n%endif\n<td class=\'${r}\'>\n%if result:\n${result.outcome[:1]} ${result.get_run_time().strftime(\'%M:%S:%f\')}\n%else:\n-\n%endif\n</td>\n%endfor\n%if test_id in test_details:\n<td class="test_id"><a href="${test_id}.html">${test_id}</a></td></tr>\n%else:\n<td class="test_id">${test_id}</td></tr>\n%endif\n%endfor\n<tr>\n%for total in totals:\n<td class=\'total\'>${time2datetime(total).strftime(\'%H:%M:%S:%f\')}</td>\n%endfor\n<td class=\'total\'>Total run-time</td>\n</tr>\n</table>\n'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
archive_db
= 'CREATE GENERATOR GEN_ANNOTATIONS_ID;\n\nSET GENERATOR GEN_ANNOTATIONS_ID TO 0;\n\nCREATE GENERATOR GEN_ANN_TYPES_ID;\n\nSET GENERATOR GEN_ANN_TYPES_ID TO 0;\n\nCREATE GENERATOR GEN_OUTCOMES_ID;\n\nSET GENERATOR GEN_OUTCOMES_ID TO 0;\n\nCREATE GENERATOR GEN_RUNS_ID;\n\nSET GENERATOR GEN_RUNS_ID TO 0;\n\nCREATE GENERATOR GEN_TESTS_ID;\n\nSET GENERATOR GEN_TESTS_ID TO 0;\n\nCREATE TABLE ANN_TYPES (\n PK BIGINT NOT NULL,\n NAME VARCHAR(70) NOT NULL);\n\nCREATE TABLE ANNOTATIONS (\n PK BIGINT NOT NULL,\n ANN_TYPE_ID BIGINT NOT NULL,\n OUTCOME_ID BIGINT NOT NULL,\n ANNOTATION BLOB SUB_TYPE 1 SEGMENT SIZE 80);\n\nCREATE TABLE OUTCOMES (\n PK BIGINT NOT NULL,\n RUN_ID BIGINT NOT NULL,\n TEST_ID BIGINT NOT NULL,\n KIND CHAR(3) CHARACTER SET ASCII NOT NULL,\n OUTCOME CHAR(1) CHARACTER SET ASCII NOT NULL,\n RUN_TIME TIME);\n\nCREATE TABLE RUNS (\n PK BIGINT NOT NULL,\n CREATED TIMESTAMP NOT NULL,\n VER VARCHAR(15) CHARACTER SET ASCII NOT NULL,\n BUILD BIGINT,\n ARCH CHAR(2) CHARACTER SET ASCII NOT NULL,\n PLANTFORM CHAR(1) CHARACTER SET ASCII NOT NULL,\n CPU VARCHAR(7) CHARACTER SET ASCII NOT NULL,\n PERSON_ID CHAR(2) NOT NULL,\n PERSON VARCHAR(25) NOT NULL,\n SEQ INTEGER NOT NULL,\n DESCRIPTION VARCHAR(30));\n\nCREATE TABLE TESTS (\n PK BIGINT NOT NULL,\n NAME VARCHAR(300) NOT NULL\n);\n\nALTER TABLE ANNOTATIONS ADD CONSTRAINT PK_ANNOTATIONS PRIMARY KEY (PK);\n\nALTER TABLE ANN_TYPES ADD CONSTRAINT PK_ANN_TYPES PRIMARY KEY (PK);\n\nALTER TABLE OUTCOMES ADD CONSTRAINT PK_OUTCOMES PRIMARY KEY (PK);\n\nALTER TABLE RUNS ADD CONSTRAINT PK_RUNS PRIMARY KEY (PK);\n\nALTER TABLE TESTS ADD CONSTRAINT PK_TESTS PRIMARY KEY (PK);\n\nALTER TABLE ANNOTATIONS ADD CONSTRAINT FK_ANNOTATIONS_1 FOREIGN KEY (ANN_TYPE_ID) REFERENCES ANN_TYPES (PK) ON DELETE CASCADE ON UPDATE CASCADE;\n\nALTER TABLE ANNOTATIONS ADD CONSTRAINT FK_ANNOTATIONS_2 FOREIGN KEY (OUTCOME_ID) REFERENCES OUTCOMES (PK) ON DELETE CASCADE ON UPDATE CASCADE;\n\nALTER TABLE OUTCOMES ADD CONSTRAINT FK_OUTCOMES_1 FOREIGN KEY (RUN_ID) REFERENCES RUNS (PK) ON DELETE CASCADE ON UPDATE CASCADE;\n\nALTER TABLE OUTCOMES ADD CONSTRAINT FK_OUTCOMES_2 FOREIGN KEY (TEST_ID) REFERENCES TESTS (PK) ON DELETE CASCADE ON UPDATE CASCADE;\n\nCREATE INDEX RUNS_IDX1 ON RUNS (CREATED);\n\nCREATE INDEX RUNS_IDX2 ON RUNS (VER);\n\nCREATE INDEX RUNS_IDX3 ON RUNS (PLANTFORM);\n\nCREATE INDEX RUNS_IDX4 ON RUNS (CPU);\n\nCREATE INDEX RUNS_IDX5 ON RUNS (ARCH);\n\nCREATE INDEX RUNS_IDX6 ON RUNS (PERSON_ID);\n\nCREATE INDEX RUNS_IDX7 ON RUNS (BUILD);\n\nCREATE TRIGGER ANNOTATIONS_BI FOR ANNOTATIONS\nACTIVE BEFORE INSERT POSITION 0\nas\nbegin\n if (new.pk is null) then\n new.pk = gen_id(gen_annotations_id,1);\nend\n\nCREATE TRIGGER ANN_TYPES_BI FOR ANN_TYPES\nACTIVE BEFORE INSERT POSITION 0\nas\nbegin\n if (new.pk is null) then\n new.pk = gen_id(gen_ann_types_id,1);\nend\n\nCREATE TRIGGER OUTCOMES_BI FOR OUTCOMES\nACTIVE BEFORE INSERT POSITION 0\nas\nbegin\n if (new.pk is null) then\n new.pk = gen_id(gen_outcomes_id,1);\nend\n\nCREATE TRIGGER RUNS_BI FOR RUNS\nACTIVE BEFORE INSERT POSITION 0\nas\nbegin\n if (new.pk is null) then\n new.pk = gen_id(gen_runs_id,1);\nend\n\nCREATE TRIGGER TESTS_BI FOR TESTS\nACTIVE BEFORE INSERT POSITION 0\nas\nbegin\n if (new.pk is null) then\n new.pk = gen_id(gen_tests_id,1);\nend'¶ str(object=’‘) -> string
Return a nice string representation of the object. If the argument is a string, the return value is the same object.
-
fbtest.
script_runner
¶ ScriptRunner
instance.
Functions¶
-
fbtest.
archive_db_commands
()¶
-
fbtest.
time2datetime
(sec)¶
-
fbtest.
xml_safe
(value)¶ Replaces invalid XML characters with ‘?’.
-
fbtest.
escape_cdata
(cdata)¶ Escape a string for an XML CDATA section.
-
fbtest.
trim_value
(value)¶ Return string with trailing whitespaces from each line removed.
-
fbtest.
quote
(value)¶ Return properly quoted string according its content.
-
fbtest.
compare_versions
(lver, rver)¶ Compare two version strings, returns: 0 for lver = rver -1 for lver < rver 1 for lver > rver
-
fbtest.
as_unicode
(value)¶ Make sure that string is unicode. Accepts Unicode, strings encoded in UTF-8 or None and returns Unicode string or None.
-
fbtest.
as_utf8
(value)¶ Make sure that string is encoded in UTF-8. Accepts Unicode, strings or None. If paremeter is unicode, it’s returned as string encoded in UTF-8. String or None parameter is returned as is.
-
fbtest.
runProgram
(args, environment, stdin=None, **kwargs)¶ Run external program using
Popen
from subprocess module and capture stdin and stderr.Parameters: - args (list) – List of arguments for
subprocess.Popen
. First item must be specification of program to run. - environment (dict) – Dictionary of environment variables. Ignored in this version.
- stdin (string) – String to be passed as stdin to external program.
- kwarg (dict) – Dictionary of additional keyword arguments for Popen.
Returns: Tuple of (returncode, stdout, stderr)
- args (list) – List of arguments for
Classes¶
TestVersion class¶
-
class
fbtest.
TestVersion
(id, platform, firebird_version, test_type, test_script, database='New', expected_stdout='', expected_stderr='', database_name=None, backup_file=None, user_name='SYSDBA', user_password='masterkey', database_character_set=None, connection_character_set=None, page_size=None, sql_dialect=3, init_script='', resources=None, substitutions=None, qmid=None)¶ Recipe for test execution against specific Firebird version and OS platform.
Parameters: - id (string) – Test ID (dot-separated name).
- platform (string) – List of platform names separated by colon.
- firebird_version (string) – First Firebird version this test version is designed for.
- test_type (string) – Test implemetation method: “ISQL” or “Python”.
- test_script (string) – Test code.
- database (string) – Database usage specification: None, “New”, “Existing” or “Restore”.
- expected_stdout (string) – Expected STDOUT content.
- expected_stderr (string) – Expected STDERR content.
- database_name (string) – Database file name.
- backup_file (string) – Backup file name.
- user_name (string) – User name.
- user_password (string) – User password.
- database_character_set (string) – Character set for database.
- connection_character_set (string) – Character set for connection.
- page_size (string) – Page size for database.
- sql_dialect (integer) – SQL dialect for database.
- init_script (string) – Test initialization script.
- resources (list) – List of
Resource
names. - substitutions (list) – List of substitution specifications.
- qmid (string) – Test ID from old system (QMTest).
-
as_expression
()¶ Return recipe data as string definition of Python dictionary. This string could be evaluated back to Python dictionary and passed to
TestVersion
constructor to recreate the test version instance. String is encoded in UTF-8 if necessary.Note
Only attributes that haven’t DEFAULT values are included.
-
get_platforms
()¶ Returns platforms supported by this test version as list of platform names.
-
run
(context, result)¶ Execute the recipe.
Important
Test run outcome is stored in
Result
instance.Parameters:
-
FIELDS
= ['id', 'qmid', 'firebird_version', 'platform', 'database', 'database_name', 'backup_file', 'user_name', 'user_password', 'database_character_set', 'connection_character_set', 'page_size', 'sql_dialect', 'init_script', 'test_type', 'test_script', 'expected_stdout', 'expected_stderr', 'resources', 'substitutions']¶
Test class¶
-
class
fbtest.
Test
(id, title='', description='', tracker_id='', min_versions=None, versions=None, qmid=None)¶ Test definition.
Can contain multiple
TestVersion
recipes for test execution against different Firebird versions on various OS platforms.Parameters: - id (string) – Test ID (dot-separated name).
- title (string) – Test title.
- description (string) – test description.
- tracker_id (string) – JIRA entry ID this tests relates to.
- min_versions (string) – List of minimal engine versions supported by test (separated by semicolon).
- versions (list) – List of disctionaries with data for creation of
TestVersion
instances. - qmid (string) – Test ID from old system (QMTest).
-
add_version
(version)¶ Add
TestVersion
instance to test.Parameters: version – TestVersion
instance.
-
as_expression
()¶ Return test data as string definition of Python dictionary. This string could be evaluated back to Python dictionary and passed to
Test
constructor to recreate the test instance. String is encoded in UTF-8 if necessary.Note
Only attributes that haven’t DEFAULT values are included.
-
get_name
()¶ Return test name.
Test name is last part of test ID.
-
get_version_for
(platform, version)¶ Return test recipe suitable for specified platform and Firebird version.
Parameters: Returns: TestVersion
or None.
-
FIELDS
= ['id', 'qmid', 'tracker_id', 'title', 'description', 'min_versions', 'versions']¶
Resource class¶
-
class
fbtest.
Resource
(id)¶ Base class for test resources.
Parameters: id (string) – Resource ID. -
as_expression
()¶ Return resource data as string definition of Python dictionary. This string could be evaluated back to Python dictionary and passed to
Resource
constructor to recreate the resource instance. String is encoded in UTF-8 if necessary.
-
cleanup
(result)¶ Resource finalization
Nothing in this class.
-
classmethod
create
(kind, **kwargs)¶ Class method to create right
Resource
instance from parameters.Parameters: kind (string) – Resource class specification.
-
fail_and_annotate_streams
(result, outcome, program_name, cause, stdout=None, stderr=None, annotate={})¶ Helper method to note fail of resource setup/cleanup in
Result
instance.Parameters:
-
setup
(context, result)¶ Resource initialization.
Nothing in this class.
-
UserResource class¶
-
class
fbtest.
UserResource
(id, user_name, user_password)¶ Test resource that provides Firebird USER.
Parameters: -
cleanup
(context, result)¶ Remove specified Firebird user via GSEC.
-
setup
(context, result)¶ Create Firebird user with specified name and password via GSEC.
-
FIELD
= ['id', 'user_name', 'user_password']¶
-
Suite class¶
-
class
fbtest.
Suite
(name, path=None, parent=None)¶ Container for tests and sub-suites to organize tests in hierarchical structure.
Parameters: -
clear
()¶ Delete all tests and sub-suites from instance.
-
get_id
()¶ Return suite ID.
Consists from suite names from root to this one separated by dot.
-
get_parent
()¶ Return parent suite or None
-
get_tests
()¶ Return all tests for this suite (including tests from sub-suites).
-
load
()¶ Load tests and sub-suites from
path
.
-
save_test
(test_id)¶ Save test on disk.
Parameters: test_id – Test ID.
-
Repository class¶
Archive class¶
-
class
fbtest.
Archive
(archive_path)¶ Run results archive.
Archive is structured in subdirectories. Each directory is named after Firebird version and contains result collection dump (pickle) files with name:
<number_of_tests_executed>-<platform><cpuarch>-<fbarch>-<person-id>-<sequence>.trf
Parameters: archive_path (string) – Directory with archive. -
delete
(result)¶ Remove specified run results from archive.
Parameters: result – File path in archive or RunResult
instance stored in archive.
-
get_archive_file_path
(results)¶ Get File path for archived results. :param
RunResult
results: Run results. :returns: File path.
-
get_archive_filename
(results)¶
-
list_filenames
()¶ Get list of result files in archive. :returns: Sorted list of filenames (including FB version subdirectory name).
-
retrieve
(version=None)¶ Get list of run results from archive.
Parameters: version (string) – Returns only results for specified Firebird version. Returns: List of RunResult
instances.
-
store
(results)¶ Store run results into archive. :param
RunResult
results: Run results.
-
Result class¶
-
class
fbtest.
Result
(kind, id, outcome='PASS', annotations={})¶ Result from test run or resource setup/cleanup.
Note
Also acts as dictionary of annotations.
Parameters: -
error
(cause=None, annotations={})¶ Set ERROR outcome with specified cause and annotations.
Parameters:
-
fail
(cause=None, annotations={})¶ Set FAIL outcome with specified cause and annotations.
Parameters:
-
get
(key, default=None)¶
-
get_cause
()¶ Return cause.
-
get_elapsed
()¶ Return test run time as (float) number of secods.
-
get_run_time
()¶ Return test run time as datetime.time.
-
has_key
(key)¶
-
items
()¶
-
keys
()¶
-
note_exception
(exc_info=None, cause=None)¶ Set outcome to ERROR and annotate with exception info and traceback.
Parameters: - exc_info – Exception info or None to use current exception info.
- cause (string) – Exception cause or None for default message.
-
set_cause
(cause)¶ Set cause.
-
set_outcome
(outcome, cause=None, annotations={})¶ Set Result outcome.
Parameters:
-
CAUSE
= 'cause'¶
-
END_TIME
= 'end_time'¶
-
ERROR
= 'ERROR'¶
-
EXCEPTION
= 'exception'¶
-
FAIL
= 'FAIL'¶
-
PASS
= 'PASS'¶
-
RESOURCE
= 'resource'¶
-
RESOURCE_CLEANUP
= 'resource_cleanup'¶
-
RESOURCE_SETUP
= 'resource_setup'¶
-
SKIPPED
= 'SKIPPED'¶
-
START_TIME
= 'start_time'¶
-
TEST
= 'test'¶
-
TRACEBACK
= 'traceback'¶
-
UNTESTED
= 'UNTESTED'¶
-
kinds
= ['resource_setup', 'resource_cleanup', 'test']¶
-
outcomes
= ['ERROR', 'FAIL', 'UNTESTED', 'PASS', 'SKIPPED']¶
-
RunResults class¶
-
class
fbtest.
RunResults
¶ Collection of test/resource Results.
Designed to hold Results from single QA run. Acts also as dictionary of Results with test/resource ID as key and
Result
object as value.-
add
(result)¶ Add result
-
clear
()¶ Remove all results.
-
get
(key, default=None)¶
-
get_error_count
()¶ Return number of ERROR outcomes
-
get_errors
()¶ Return list of Results with ERROR outcome
-
get_fail_count
()¶ Return number of FAIL outcomes
-
get_fails
()¶ Return list of Results with FAIL outcome
-
get_outcomes
()¶ Get list outcomes from stored Results
-
get_pass_count
()¶ Return number of PASS outcomes
-
get_passes
()¶ Return list of Results with PASS outcome
-
get_skipped
()¶ Return list of Results with SKIPPED outcome
-
get_skipped_count
()¶ Return number of SKIPPED outcomes
-
get_untested
()¶ Return list of Results with UNTESTED outcome
-
get_untested_count
()¶ Return number of UNTESTED outcomes
-
has_key
(key)¶
-
items
()¶
-
keys
()¶
-
classmethod
load
(filename)¶ Load results from file.
Parameters: filename (string) – Filename with pickled Result
.
-
print_report
()¶
-
print_summary
()¶ Print results summary to stdout.
-
values
()¶
-
Runner class¶
-
class
fbtest.
Runner
(repository)¶ QA Execution engine.
Parameters: repository – Repository
instance.-
run
(test_list=None, skip_list=None, verbosity=1, results=None, no_summary=False, expectations=None)¶ Run tests.
Parameters: Returns: RunResults
object with results.Warning
QA environment must be properly configured!
-
set_person
(person)¶ Set QA person identification.
Parameters: person (string) – QA person name. Sets person_name and person_id attributes: person_name = person in lowercase person_id = first two letters from uppercased person
If person is “Unknown”, the person_is is set to “XX”.
-
set_target
(arch, host, bin_dir=None, password='masterkey', client_lib=None)¶ Configures the QA environment to run on specified Firebird installation.
Parameters:
-
tempdir
¶
-
ScriptRunner class¶
-
class
fbtest.
ScriptRunner
¶ Helper class to organize script running code. Instantiated as singleton instance script_runner.
-
analyze
(filenames, output_dir, diffs_only=False)¶ Analyze test run results and produce HTML output.
Parameters:
-
annotation_filter
(annotations)¶ Filters out annotations we don’t want to compare.
-
cmd_archive_delete
(options)¶ Called by
run_archive()
for command execution.
-
cmd_archive_list
(options)¶ Called by
run_archive()
for command execution.
-
cmd_archive_retrieve
(options)¶ Called by
run_archive()
for command execution.
-
cmd_archive_save
(options)¶ Called by
run_archive()
for command execution.
-
cmd_db_create
(options)¶ Called by
run_database()
for command execution.
-
cmd_db_import
(options)¶ Called by
run_database()
for command execution.
-
cmd_update_repository
(options)¶ Called by
run_update()
for command execution.
-
cmd_update_results
(options)¶ Called by
run_update()
for command execution.
-
compare_results
(r1, r2)¶ Compare two results for single test. Compares
kind
,outcome
andannotations
.Parameters: Returns: True if both results are equal.
-
get_repository
(options)¶ Connect to local or remote Test Repository.
Parameters: options – Command-line options. Returns: Repository
instance.
-
get_result_filenames
(directory)¶ Return list of Result file filenames stored in directory.
Parameters: directory (string) – directory with result files. Returns: List with filenames.
-
get_run_tag
(platform, cpuarch, arch, sequence)¶ Return string that could be used as key to group tests by platform, cpu, FB architecture and run sequence.
-
get_svn_login
(realm, username, may_save)¶ Get Subversion login credentials from user.
-
print_analysis
(version, results, tests, test_details, test_order, output_dir, diffs_only)¶ Create HTML files with test run analysis.
Parameters: - version (string) – Firebird version.
- result (
RunResults
) – Run results. - tests (dict) – Dictionary of all tests found in results; Test ID: list of results.
- test_details (dict) – Dictionary with test run results: Run tag: Result.
- test_order (list) – List that define test order in output.
- output_dir (string) – Directory for HTML output files.
-
print_errors
(result, cause=False, detail=False)¶ Print IDs of tests that ended with ERROR.
Parameters: - result (
RunResults
) – Run results. - cause (bool) – Print error cause.
- detail (bool) – Print error details.
- result (
-
print_fails
(result, cause=False, detail=False)¶ Print IDs of tests that ended with FAIL.
Parameters: - result (
RunResults
) – Run results. - cause (bool) – Print fail cause.
- detail (bool) – Print fail details.
- result (
-
print_result_info
(result, filename)¶ Print information from result file.
Parameters: - result (
RunResults
) – Run results. - filename (string) – Results filename.
- result (
-
print_skipped
(result, cause=False)¶ Print IDs of tests that ended with SKIPPED.
Parameters: - result (
RunResults
) – Run results. - cause (bool) – Print cause.
- result (
-
print_untested
(result, cause=False)¶ Print IDs of tests that ended with UNTESTED.
Parameters: - result (
RunResults
) – Run results. - cause (bool) – Print cause.
- result (
-
run_analyze
(options)¶ Called by
run_analyze()
for command execution.
-
run_server
(options)¶ Called by
run_server()
for command execution.
-
run_tests
(options)¶ Called by
run_tests()
for command execution.
-
run_view
(options)¶ Called by
run_view()
for command execution.
-
svn_notify
(event)¶ Notification callback from pysvn.
-
platform_table
= {'macos': 'Mac', 'hp-ux': 'HPUX', 'solaris': 'Solaris', 'windows': 'Win', 'freebsd': 'BSD', 'linux': 'Linux'}¶
-
Script Functions¶
-
fbtest.
run_tests
()¶ CLI Script function for test execution.
This is a ‘main’ function called by fbt_run script. Runs all tests in test reporsitory or specified test/suite. Result from test run is stored in current directory:
results.trf
: PickledRunResults
instance. PickledRunResults
instance is also stored in ‘archive’.results.xml
: XUNIT XML report (when -x or –xunit is specified).usage: fbt_run [options] [suite_or_test_name]
-
-v
,
--verbose
¶
Be more verbose
-
--verbosity
=NUM
¶ Set verbosity; –verbosity=2 is the same as -v
-
-q
,
--quiet
¶
Be less verbose
-
-b
<directory>
,
--bin-dir
=<directory>
¶ Location of Firebird binary tools (like gbak etc.).
-
-d
<directory>
,
--db-dir
=<directory>
¶ Location for temporary databases used by tests.
-
-k
<name_or_file>
,
--skip
=<name_or_file>
¶ Suite or test name or name of file with suite/test names to skip.
-
--archive
¶
If specified, last run result file is copied to archive.
-
--rerun
¶
If specified, runs only tests that don’t PASSed in last run.
-
--remote
¶
If specified, connects to remote fbtest server and runs specified tests on it.
-
-u
,
--update
¶
If specified, updates last run results with re-run results.
-
-w
<password>
,
--password
=<password>
¶ SYSDBA password. (default ‘masterkey’)
-
-o
<machine>
,
--host
=<machine>
¶ Firebird or fbtest host machine identification. (default ‘localhost’)
-
-p
<person>
,
--person
=<person>
¶ QA person name. First two letters are used as person ID.
-
-a
<arch>
,
--arch
=<arch>
¶ Firebird architecture (‘SS’, ‘CS’ or ‘SC’). (default ‘SS’ - SuperServer)
-
-s
<number>
,
--sequence
=<number>
¶ Run sequence number. Important to disctinguish repeated runs af the same tests on single ‘target’. Used by fbt_analyze. (default ‘1’)
-
-x
,
--xunit
¶
Provides test results also in the standard XUnit XML format.
-
-e
,
--expect
¶
Test results file to be used as expeted outcomes
-
-
fbtest.
run_server
()¶ CLI Script function for test execution.
This is a ‘main’ function called by fbt_server script.
usage: fbt_server [options]
-
-b
<directory>
,
--bin-dir
=<directory>
¶ Location of Firebird binary tools (like gbak etc.).
-
-d
<directory>
,
--db-dir
=<directory>
¶ Location for temporary databases used by tests.
-
-w
<password>
,
--password
=<password>
¶ SYSDBA password. (default ‘masterkey’)
-
-o
<machine>
,
--host
=<machine>
¶ Firebird host machine identification. (default ‘localhost’)
-
-p
<person>
,
--person
=<person>
¶ QA person name. First two letters are used as person ID.
-
-a
<arch>
,
--arch
=<arch>
¶ Firebird architecture (‘SS’, ‘CS’ or ‘SC’). (default ‘SS’ - SuperServer)
-
--register
¶
If specified, automatically registers RPyC slave server.
-
-
fbtest.
run_analyze
()¶ CLI Script function for test analysis.
This is a ‘main’ function called by fbt_analyze script. Reads result file or all result files in specified directory and writes HTML report in current or specified directory.
usage: fbt_analyze [options] <file_or_directory>
-
-o
<directory>
,
--output
=<directory>
¶ Location where HTML report should be stored.
-
-d
,
--difs-only
¶
Show only diffs on detail pages.
-
-
fbtest.
run_update
()¶ CLI Script function to update metadata in Result file(s).
This is a ‘main’ function called by fbt_update script. Reads result file or all result files in specified directory and updates metadata with specified value(s).
usage:
fbt_update {result,repository} [options] result Change result file metadata. repository Update Test Repository.
Options for fbt_update result:
-
name
¶
Results file or directory with result files.
-
-a
<arch>
,
--arch
=<arch>
¶ Firebird architecture (‘SS’, ‘CS’ or ‘SC’).
-
-p
<person>
,
--person
=<person>
¶ QA person name. First two letters are used as person ID.
-
-s
<number>
,
--sequence
=<number>
¶ Run sequence number. Important to disctinguish repeated runs of the same tests on single ‘target’. Used by fbt_analyze.
Options for fbt_update repository:
None.-
-
fbtest.
run_view
()¶ CLI Script function to update metadata in Result file(s).
This is a ‘main’ function called by fbt_view script. Reads result file or all result files in specified directory and prints information about stored results.
usage: fbt_view <file_or_directory>
-
-x
,
--xunit
¶
Store processed test results in the standard XUnit XML format.
-
-c
,
--cause
¶
Print cause of fails and errors.
-
-d
,
--details
¶
Print details for fails and errors.
-
-
fbtest.
run_archive
()¶ CLI Script function to maintain archive of Result file(s).
This is a ‘main’ function called by fbt_archive script.
usage:
fbt_update {list,save,retrieve,delete} [options] list List result(s) in archive. save Save result(s) to archive. retrieve Retrieve result(s) from archive. delete Delete result(s) from archive.
-
--remote
¶
If specified, connects to remote fbtest server (and archive).
-
-o
<machine>
,
--host
=<machine>
¶ Remote fbtest host machine identification. (default ‘localhost’)
Options for fbt_archive list:
None.Options for fbt_archive save:
-
name
¶
Results file.
Options for fbt_archive retrieve:
-
-v
<version_number>
,
--version
=<version_number>
¶ Retrieve results only for specified Firebird version.
-
-c
,
--current
¶
Retrieve results only for currently tested Firebird version.
-
-o
<directory>
,
--output
=<directory>
¶ Location where result file(s) should be copied.
-
-p
<person>
,
--person
=<person>
¶ Retrieve results only for specified QA person name. First two letters are used as person ID.
-
-a
<arch>
,
--arch
=<arch>
¶ Retrieve results only for specified Firebird architecture (‘SS’, ‘CS’ or ‘SC’).
-
-s
<number>
,
--sequence
=<number>
¶ Retrieve results only for specified run sequence number.
Options for fbt_archive delete:
-
-v
<version_number>
,
--version
=<version_number>
¶ Delete results only for specified Firebird version.
-
-c
,
--current
¶
Delete results only for currently tested Firebird version.
-
-p
<person>
,
--person
=<person>
¶ Delete results only for specified QA person name. First two letters are used as person ID.
-
-a
<arch>
,
--arch
=<arch>
¶ Delete results only for specified Firebird architecture (‘SS’, ‘CS’ or ‘SC’).
-
-s
<number>
,
--sequence
=<number>
¶ Delete results only for specified run sequence number.
-
-
fbtest.
run_database
()¶ CLI Script function to maintain database archive of Result file(s).
This is a ‘main’ function called by fbt_db script.
usage:
fbt_db [-h] [-w PASSWORD] [-o HOST] [-d DATABASE] {import,create} ... optional arguments: -h, --help show this help message and exit -w PASSWORD, --password PASSWORD SYSDBA password -o HOST, --host HOST Firebird host machine identification -d DATABASE, --database DATABASE Archive database name Commands: {import,create} Use <command> --help for more information about command. create Creates archive database. import Import result(s) to database.
-
-o
<machine>
,
--host
=<machine>
¶ Firebird machine identification. (default ‘localhost’)
-
-d
<filename>
,
--database
=<filename> (`default 'fbtarch.fdb'`)
¶ FBTest archive database name.
-
-w
<password>
,
--password
=<password>
¶ SYSDBA password. (default ‘masterkey’)
Options for fbt_archive create:
None.Options for fbt_archive import:
-
name
¶
Results file.
-