Compare commits

...

251 Commits

Author SHA1 Message Date
gaasedelen 0350b0c538 Follow-up to #157: unsubscribe torn-down coverage overview refresh callback 2026-02-13 19:39:44 -05:00
gaasedelen 88051b0652 Follow-up to #157: keep IDA8/9.1 sip path and bump dev version to 0.9.4 2026-02-13 18:56:11 -05:00
gaasedelen 1d7e4b321c Merge PR #157: IDA 9.2/9.3 compatibility fixes 2026-02-13 18:55:54 -05:00
raptor 602bb611a4 Fix some issues in IDA 9.1 2025-09-09 14:04:28 +02:00
raptor c5cc03d7dc Improve IDA 9.2 support without breaking previous supported versions 2025-09-09 13:34:47 +02:00
raptor 720b036a9d Start working on IDA 9.2 port 2025-09-09 11:43:22 +02:00
raptor b8ff268c86 IDA 9.x SDK fix 2025-08-28 17:47:43 +02:00
gaasedelen 88ceac7b19 tweak to make lighthouse copy *all* in-box theme files to the user theme directory... 2024-02-06 13:59:33 -05:00
gaasedelen 9a56463103 make update dialog look a little less janky 2024-02-06 13:30:54 -05:00
gaasedelen 562595be9b Merge branch 'develop' 2024-02-05 13:05:19 -05:00
gaasedelen eeb16a5f81 final fixes, ticking version numbers 2024-02-05 13:03:41 -05:00
gaasedelen 84eeb210c6 minor cleanup, finalizing for release 2024-02-02 19:28:55 -05:00
gaasedelen f944b62cc3 Update Intel pin build scripts for pin-3.30-98830 (MSVC) 2024-02-02 19:06:33 -05:00
gaasedelen 1b883377d6 switch database metadata instruction listing to a set, dramatically improving coverage loading perf in several cases, fixing #128 2024-02-02 17:31:56 -05:00
gaasedelen 146eb4fd2d fix bug where 'X' column in combobox was super wide/fat under PySide6 for some reason 2024-02-02 17:03:08 -05:00
gaasedelen de2704b680 Coverage Xref right click UI action rewrite for binja 2024-02-02 16:38:04 -05:00
gaasedelen 96df2c5a1f adds "long night" theme by ioncodes 2024-02-02 12:19:31 -05:00
gaasedelen 9c579c6e9d fix #131, hopefully improves dpi issues cross platform 2024-02-02 11:55:05 -05:00
gaasedelen ee7ac30f14 fix binary ninja "suspicious" warnings 2024-02-02 11:32:04 -05:00
gaasedelen 5f8fa02de2 fix function/symbol rename event hooks in binja 2024-02-02 11:32:04 -05:00
Thomas Dupuy 87defff5cd FIx small typo. 2024-02-01 20:31:24 -05:00
gaasedelen e1438159da fixes coverage xref compatability issues (timestamp crash) with binja / PySide6 per #123 2024-02-01 20:25:08 -05:00
gaasedelen 856768d129 fix clipboard compat issue exposed by PySide6 2024-02-01 20:08:26 -05:00
gaasedelen 7d67e41773 Fixes issue raised in #129 while retaining compatability and functionality across IDA / binja. 2024-02-01 19:54:30 -05:00
gaasedelen 8f3c23bbb6 maintain compatability with IDA 2024-02-01 18:50:58 -05:00
Alexandre Brenner 4992d4b731 Fix ValueError
Only tested on Windows
2024-02-01 18:50:58 -05:00
Alexandre Brenner 550c476e81 Fix TypeError
`TypeError: BasicBlock._create_instance() takes 2 positional arguments but 3 were given`
Only tested on Windows
2024-02-01 18:50:58 -05:00
mishap mishap bba8d91253 Allow partial module whitelist match 2024-01-18 20:49:25 -05:00
gaasedelen 0e0e08e564 normalize code/comments of drcov 3 fix 2024-01-18 19:47:48 -05:00
PavelBlinnikov 6912018b89 add support for drcov version 3 2024-01-18 19:47:48 -05:00
Moritz Schloegel f4642e8b4b fix typo in gradient 2022-02-17 14:29:27 -05:00
Moritz Schloegel 6e1dcb8d40 convert float to int as expected by Qt API functions (fixes #116) 2022-02-17 14:29:27 -05:00
Alexander Taylor 973b40e6d0 Fix for func possibly being None. 2022-01-10 11:36:32 -05:00
gaasedelen e3d636a3f4 adds support for tracking, viewing, and dumping orphan coverage (#108) 2021-10-05 22:48:37 -04:00
gaasedelen 7245a2d2c4 Merge branch 'develop' 2021-08-10 16:41:04 -04:00
gaasedelen 701952d83f add x86 supprt to tenet reader 2021-08-10 16:39:05 -04:00
gaasedelen 9ed0ccf528 updated readme, version number 2021-08-10 16:35:18 -04:00
gaasedelen ae1072a04b improves background color detection logic 2021-08-09 16:43:18 -04:00
gaasedelen 4c19fee630 bugfix when loading executables with unicode names on Python 2.7 2021-08-09 16:42:47 -04:00
gaasedelen d02fbf13d9 adds support for loading text-based Tenet traces 2021-08-06 18:45:51 -04:00
_yrp 19eddd6470 Lower block trace confidence (#105)
Depending on how blocks are traced, on some targets 0.9 is too high. Specifically, some emulator based tracers might consider the first instruction after a return to be a new "block" -- this will inflate the number of instructions with coverage path the threshold, even though the trace should still be considered block cov.
2021-08-06 18:41:41 -04:00
gaasedelen 09a95061ed reduce banner print 2021-08-03 16:47:34 -04:00
gaasedelen 94759790de font tweaks again 2021-08-03 16:46:56 -04:00
gaasedelen 374ca999f8 maybe fixes qt conflict issues? (issue #100) 2021-08-03 12:31:10 -04:00
gaasedelen b3c8683110 font fixes for macos...? for the 10th time...? 2021-07-30 09:03:40 -04:00
gaasedelen aa71810bf7 PySide6 & Python2 Compat :-X 2021-07-30 09:03:11 -04:00
gaasedelen b0f798a3d9 a bit of cleanup for binja plugin manager stuff 2021-07-29 10:42:45 -04:00
gaasedelen 85b08a9611 spindown lighthouse contexts in binja 2021-07-29 10:33:00 -04:00
gaasedelen f2031f921f a bit of housekeeping on drcov.py 2021-07-29 06:08:14 -04:00
Jordan 8e98a74eee Qt6 Updates for Binja + Binja plugin manager compatability (#106)
* initial pyside6 support and testing binary ninja plugin installation

* small tweak to fix plugin working with internal test builds
2021-07-29 05:47:36 -04:00
gaasedelen 85818f5b9b Merge pull request #98 from angelystor/master
Added DrCov module table version 5 support
2021-04-09 19:41:08 -04:00
Angelystor 8434c80e18 Added DrCov module table version 5 support which the latest DynamoRIO outputs 2021-04-08 18:55:09 +08:00
DrWhax a25e77764a Fixes for fixing the PIN build for all operating systems (#94)
* remove unneeded std namespace

* remove Wno-aligned-new

* This makes the build work again, tested on VC2015 community edition on
Windows Server 2019.

Credit for these changes go to: https://github.com/HongThatCong/Lighthouse-CodeCoverage-build
2020-12-26 21:57:41 -05:00
gaasedelen 91e427e7ec Merge branch 'develop' 2020-09-02 14:57:02 -04:00
gaasedelen 1bad74fc50 updates version string, readme tweaks 2020-09-02 14:55:44 -04:00
gaasedelen 68325095ee added the ability to suppress the module selector dialog 2020-08-31 21:58:02 -04:00
gaasedelen a50298a9d9 fix crash when ida is not shipped with ida_hexrays (issue #88) 2020-08-31 21:55:50 -04:00
gaasedelen 18ea9bbc69 disable logging by default, as things seem stable... 2020-07-28 22:30:08 -04:00
gaasedelen 5818ffe3fc trim filepaths off a coverage file's 'module names' (if present) 2020-07-28 22:29:40 -04:00
gaasedelen 5258d583c7 fix regression introduced in the shared note / painting fix from a commit or two prior 2020-07-28 22:09:52 -04:00
gaasedelen 5480afe646 renamed plugin dir to plugins for easier dev 2020-07-28 19:45:58 -04:00
gaasedelen 7a1abaeaa1 fixes a bug where the wrong nodes could get painted in IDA if a 'shared' node was executed 2020-07-04 22:12:55 -04:00
gaasedelen 1cf63ac925 Update metadata.py
save confirmed edges after computing cyclomatic complexity
2020-05-12 01:25:27 -04:00
gaasedelen 2bb4b9ebf3 vastly improves Lighthouse pintool performance for larger workloads 2020-05-04 05:35:02 -04:00
gaasedelen 710b13f38e Merge branch 'develop' 2020-04-23 06:00:37 -04:00
gaasedelen 69a595a875 updates readme and version # 2020-04-23 05:55:40 -04:00
gaasedelen 4661517cb3 hack to hide binja featuremap by default, since it will probably collide with the coverage overview dock position :-X 2020-04-23 05:03:09 -04:00
gaasedelen 3f0cfa856e bugfix where paint wouldn't fully refresh after changing themes 2020-04-23 04:20:38 -04:00
gaasedelen 8b9382d414 document the CoverageFile format a bit...... 2020-04-22 01:17:29 -04:00
gaasedelen a0c77f0f72 lower the font size a bit for macos 2020-04-21 23:13:44 -04:00
gaasedelen 22d48fa52b more robust path creation 2020-04-20 19:42:24 -04:00
gaasedelen 8cbfffe2e0 actually detect when IDA is is batch..... 2020-04-20 19:42:06 -04:00
gaasedelen 1df982ca4c ensure lighthouse stays disabled while using disassemblers headlessly 2020-04-20 18:05:16 -04:00
gaasedelen 859b994bf7 improve styling cross-platform/DPI 2020-04-20 15:42:08 -04:00
gaasedelen b610b1ee32 fixes bug where combobox would immediately close on binja linux 2020-04-20 04:56:26 -04:00
gaasedelen 33ef4e47a2 fix bug where drcov parser could fail to get correct filename from a crossplatform path... 2020-04-20 04:55:31 -04:00
gaasedelen 4eaca66caa fix bug where refresh could run twice when opening cov overview 2020-04-20 03:17:42 -04:00
gaasedelen 4e8f5d3bbc improve table header styling, dialogs 2020-04-20 03:17:03 -04:00
gaasedelen a943580b2c improved warning dialogs a bit... 2020-04-19 20:52:43 -04:00
gaasedelen 112e5d6a3b fix partial painting in binja... 2020-04-19 18:00:32 -04:00
gaasedelen 75572aed33 fix so the 'prefix' delim does not show in the table / matches what IDA renders... 2020-04-19 18:00:15 -04:00
gaasedelen 7c1573bfd6 cleanup of TODO's 2020-04-19 02:33:56 -04:00
gaasedelen feb83fc5d5 re-enable headless access via lighthouse.get_context(...) 2020-04-18 23:13:23 -04:00
gaasedelen 3f33c3cb45 enable live rebasing (at least... for IDA) 2020-04-18 22:15:57 -04:00
gaasedelen 9c0ecbc81e a few fixes and tweaks for robustness, performance 2020-04-18 20:21:50 -04:00
gaasedelen a0367a85da refresh hexrays views automatically 2020-04-18 03:29:09 -04:00
gaasedelen 4708422c6a improve the stability of the IDA painter 2020-04-18 02:19:42 -04:00
gaasedelen a789220b56 improve metadata collection 2020-04-17 23:46:34 -04:00
gaasedelen bc77c0ece2 wrap force clearing with a waitbox... 2020-04-17 23:01:27 -04:00
gaasedelen 4d94680b94 better resource cleanup when unloading (helps IDA close faster...) 2020-04-17 14:01:04 -04:00
gaasedelen b2c6695042 improve double click jump precision to go to first block with coverage in function 2020-04-17 02:03:03 -04:00
gaasedelen c2ceb47b84 totally ovehaul painting to support notion of 'streaming' 2020-04-17 01:38:40 -04:00
gaasedelen 0d52ef5068 misc fixes / tweaks 2020-04-17 01:37:46 -04:00
gaasedelen 61b8fb7668 better handling of subsystem lifetimes 2020-04-17 01:37:21 -04:00
gaasedelen 68f8c884f9 make theme util function more accessible 2020-04-16 23:46:17 -04:00
gaasedelen a0b375064a improve painting perf 2020-04-15 21:49:03 -04:00
gaasedelen c5dbfb1062 more verbose painter output 2020-04-13 21:11:53 -04:00
gaasedelen ea8cd61122 improve robustness of update check 2020-04-13 20:52:35 -04:00
gaasedelen 4aa116580e minor bugfixs / QOL 2020-04-13 10:04:53 -04:00
gaasedelen 6e6084a058 adds minimal update check 2020-04-13 10:04:25 -04:00
gaasedelen 481c624b7e improve theme reactivity for binja 2020-04-13 08:59:02 -04:00
gaasedelen a3183f21f2 improve theme loading / hinting for IDA 2020-04-12 19:53:58 -04:00
gaasedelen 33b3d4e0b6 Merge branch 'fbl_binja_refactor' into develop 2020-04-12 10:40:18 -04:00
gaasedelen 7fd1e46e53 fixes instructions in partially painted nodes not getting unpainted in binja... 2020-04-12 10:35:53 -04:00
gaasedelen 4ab65c985b cleanup, fixes right click regression with IDA 2020-04-12 09:03:43 -04:00
gaasedelen 28b0ecd49c automatically build the metadata cache when the coverage overview is first shown (if the cache is not already built) 2020-04-12 08:48:06 -04:00
gaasedelen c4cf78c1dd reconcile the binja refactor with IDA 2020-04-12 04:19:04 -04:00
gaasedelen e5b29f97b7 misc cleanup of binja refactor branch & TODO's 2020-04-11 00:00:17 -04:00
gaasedelen 36a37935dc reduce debug prints in binja, manage widget view state per tab 2020-04-10 23:25:55 -04:00
gaasedelen 65fd677758 prioritize navigating to a function start, if a function start block is shared 2020-04-10 21:57:52 -04:00
gaasedelen aa4936e269 enable priority painting for binja 2020-04-10 17:42:40 -04:00
gaasedelen 00f82a2181 only paint instructions in partially executed binja nodes 2020-04-10 16:54:20 -04:00
gaasedelen 1f7e525aeb made 'jump' case insensitive for 'sub_...' funcs in binja 2020-04-10 08:12:49 -04:00
gaasedelen 1a13b23345 fixes bug where one could not 'jump' to a renamed function 2020-04-10 08:11:52 -04:00
gaasedelen 52fb3e70d0 emancipate nodes from functions, this allows coverage to properly be computed for functions that share nodes 2020-04-10 07:46:58 -04:00
gaasedelen ff2c0d9619 switch to pprofile for line profiling 2020-04-10 05:27:08 -04:00
gaasedelen ebea88465f binja-specific perf wins 2020-04-10 03:05:37 -04:00
gaasedelen 67e5caf62d mutual perf wins for metadata caching 2020-04-10 03:03:53 -04:00
gaasedelen a6eeafe180 overhaul for proper binja integration 2020-04-08 11:42:32 -04:00
gaasedelen 8c4e29fe50 bugfix: automatically evaluate the shell expression when switching to the hot shell 2020-04-08 01:43:13 -04:00
gaasedelen 9eca228925 fixes bug that could cause one to be prompted multiple times for a composition name 2020-04-08 01:06:22 -04:00
gaasedelen 553eeb7d23 fixes grammar regression... 2020-04-08 00:43:06 -04:00
gaasedelen 914b731676 fixes bug where combobox could get stuck closed 2020-04-04 23:17:06 -04:00
gaasedelen 6571b0735f aggregate symbol was unusable in compositions 2020-04-04 19:13:34 -04:00
gaasedelen 83e3b423b9 removes range syntax from grammar, buggy & nobody uses it... 2020-04-04 19:10:58 -04:00
gaasedelen 2ed77f305d improve xref dialog for high dpi 2020-04-04 17:44:48 -04:00
gaasedelen 25ff8ed245 fix theme auto color selection bugginess 2020-04-04 16:24:27 -04:00
gaasedelen 3054246a8d bugfix for theme auto paint color selection 2020-04-04 08:30:36 -04:00
gaasedelen 5670e3116c improve xref dialog 2020-04-04 03:19:49 -04:00
gaasedelen 8a2c011636 combobox style tweaks 2020-04-04 02:37:14 -04:00
gaasedelen 6ad0af71cf fix bug when unable to find mappable intructions in cov data 2020-04-04 02:11:59 -04:00
gaasedelen 181b13d0f7 more metadata cache cleanup, robustness, QUALITY SOFTWARE 2020-04-03 21:26:22 -04:00
gaasedelen 5433cdc8e4 streamline metadata collection, allow transition from async to synchronous 2020-04-03 07:18:54 -04:00
gaasedelen 416a46b8ab small QoL tweaks 2020-04-03 07:16:00 -04:00
gaasedelen db1f4ebc68 make coverage xref menu action only appear if there is coverage loaded 2020-04-03 00:19:05 -04:00
gaasedelen c44f35e5f4 add a dark / light coverage paint variant to the themes 2020-04-02 23:55:42 -04:00
gaasedelen 7d7ee5b9f0 improve click + drag text selection in composing shell 2020-04-02 21:38:12 -04:00
gaasedelen 4d36be57c8 add a menu option to dump unmappable coverage data to the console... 2020-04-02 21:06:51 -04:00
gaasedelen 98745a09c4 a few more readme updates 2020-04-02 20:34:27 -04:00
gaasedelen 6dcb3c769c Update README.md
at least some basic updates to the readme so as to not confuse anyone trying to use the dev branch
2020-04-02 20:10:03 -04:00
gaasedelen 11d5f9e62f enforce stricter logic around fuzzy name matching #63 2020-04-02 19:19:52 -04:00
gaasedelen cad8679170 tweaks & typos, i'm tired 2020-04-02 08:13:04 -04:00
gaasedelen 410adc45a5 improve coverage xref styling 2020-04-02 08:03:59 -04:00
gaasedelen 02ea88e3d7 adds fallback selector dialog if the database's loaded module (binary) cannot be found in a coverage file. closes #63 2020-04-02 08:03:35 -04:00
gaasedelen e89a36b9d6 minor binja tweaks, to keep dev working with it for now... 2020-04-02 06:00:34 -04:00
gaasedelen f6902baf38 update drcov parser to account for multi-segment modules 2020-04-02 04:42:57 -04:00
gaasedelen 79c90db5b0 fixes bug where a leftover / mostly deleted coverage overview could get left hanging around 2020-04-02 00:39:30 -04:00
gaasedelen 93228c880b Merge branch 'fbl_themes' into develop 2020-04-02 00:03:42 -04:00
gaasedelen 4b63a0f857 Rename dark theme, again. crosses off themes from readme 2020-04-02 00:02:25 -04:00
gaasedelen fb65c06b1c some cleanup 2020-04-01 23:55:09 -04:00
gaasedelen 2af8854673 add a bit of theme validation ... 2020-04-01 23:54:57 -04:00
gaasedelen 9b85603828 rename 'classic' theme to 'synthwave' 2020-04-01 23:19:54 -04:00
gaasedelen f7ade4eaa3 enable theme change button 2020-04-01 23:18:59 -04:00
gaasedelen be7815ff06 remove hardcoded colors from parts of HTML report 2020-04-01 23:08:08 -04:00
gaasedelen a52b8d5fda split out all theme-dependent code into refreshable functions 2020-04-01 09:15:45 -04:00
gaasedelen 5ae17c85c8 add theme change callback to the palette 2020-04-01 07:32:18 -04:00
gaasedelen 2a2b646f29 minor cleanup / refactoring of palette 2020-03-31 22:43:27 -04:00
gaasedelen 9946863e41 populate user theme dir, auto-select best theme, save/load theme preference from disk, 2020-03-31 19:18:25 -04:00
gaasedelen 2d7d0d598b adds light theme 2020-03-31 02:31:16 -04:00
gaasedelen 72ec6cccf4 updates, more tweakable fields 2020-03-31 02:30:24 -04:00
gaasedelen d53e5032d0 initial theme/palette refactoring 2020-03-31 00:02:37 -04:00
gaasedelen da5942466a update dates and version number, since there won't be an v0.8.4 release... 2020-03-29 17:34:21 -04:00
gaasedelen 0ef5c9d9e1 expose the active CoverageDirector object instance for the disassembler interpreter, or other scripts to use 2020-03-29 17:31:07 -04:00
Jan Beck b1488c3086 Add compatibility with PIN 3.11 (#77)
* Add compatibility with PIN 3.11

Co-authored-by: gaasedelen <gaasedelen@users.noreply.github.com>
2020-03-29 05:36:04 -04:00
lucasg b8a996b5f1 Update README.md to provide more details on finding IDA plugin dir w/ API (#72) 2020-03-29 05:13:27 -04:00
Alexandre Maloteaux e5b9f34193 fix for ida 7.4 with python3.7 (#79)
* fix for ida 7.4 with python3.7

* fix html export too

* fix painter

* tweakd to use our own dict shims

* a few more minor fixes

Co-authored-by: gaasedelen <markus.gaasedelen@gmail.com>
2020-03-29 04:59:29 -04:00
gaasedelen 257d69594e fully deprecates IDA 6.X 2020-03-29 03:43:04 -04:00
gaasedelen 28ea6b8821 make drcov.py work natively outside of lighthouse 2020-03-29 02:43:28 -04:00
gaasedelen d93b52354f make it so that partially executed nodes do not have their whole graph node colored 2020-03-28 23:03:24 -04:00
gaasedelen e0d309025f improve lighthouse's accuracy on interleaved instructions 2020-03-25 06:22:45 -04:00
gaasedelen c57296e649 fix improper usage of traceback 2020-03-25 03:11:58 -04:00
gaasedelen c9788723ba remove unecessary use of map, can cause ambiguity if left unevaluated 2020-03-25 03:11:04 -04:00
gaasedelen 29ecafd885 fixes bug where binja could hang while building metadata 2020-03-22 03:10:27 -04:00
gaasedelen b9d514823b shady temp fix to make lighthouse mostly work with current binja 2020-03-21 04:36:28 -04:00
gaasedelen 2eab6d902c python 3 compat tweaks 2020-03-21 04:36:28 -04:00
gaasedelen 1ddffa5a19 most of the plumbing for supporting an image rebase, barring more testin... 2020-03-21 04:36:28 -04:00
gaasedelen e70c246637 make BADADDR more 'universal' 2020-03-21 04:36:28 -04:00
gaasedelen c0afc3b0c1 create 'is disassembelr busy' api 2020-03-21 04:36:28 -04:00
gaasedelen 3fd3640517 Improves coverage exception warning code 2020-03-21 04:36:28 -04:00
gaasedelen e4ecc0dafd fixes minor Qt warning for IDA 7.4 2020-03-21 04:36:28 -04:00
Jakob Pearson ac12ef74db Minor changes, to compile on Windows with VS2017 (#69)
* Update build-x64.bat

* Update build-x86.bat

* Update ImageManager.cpp

* Update CodeCoverage.cpp

* Update CodeCoverage.cpp

* Update ImageManager.cpp
2019-10-22 12:13:27 -07:00
yrp 8593e976d1 Derp, syntax... 2019-10-22 15:06:42 -04:00
yrp b505af9956 Allow blank and comments in modoff 2019-10-22 15:06:42 -04:00
yrp 7578faea82 Fix directory types
Tbh I didn't analyze this in depth -- it was a type error causing
lighthouse to bail out so I just mashed everything into a set. It seems
to work tho, so #shipit
2019-10-22 15:06:42 -04:00
Dominik Maier 7caa33df2e Fixed python3 bytesstrings (#71) 2019-10-22 15:03:17 -04:00
gaasedelen 9b1d579d3d improve disassembler logging compatibility 2019-05-18 16:56:11 -04:00
gaasedelen f6932bd8d0 adds qt_mainthread global 2019-05-17 19:17:39 -04:00
gaasedelen 80404a9f57 make disassembler version fields less annoying 2019-05-17 14:08:25 -04:00
volokitinss 366df2f5d8 Match upper case hex symbols as well (#67) 2019-05-10 12:29:52 -04:00
gaasedelen 7303d72fa2 python 3 compat 2019-04-19 19:58:29 -04:00
gaasedelen 8a656b10d3 tweak: make the function name column left aligned by default 2019-04-19 19:53:03 -04:00
gaasedelen 80d65fd584 styling consistency improvements 2019-04-19 19:52:34 -04:00
gaasedelen 8a03f035a1 improve shell / combobox styling 2019-04-17 17:15:18 -04:00
gaasedelen 9558763e52 bugfix: collapse the coverage combobox when its head is clicked in the expanded state 2019-04-16 18:24:40 -04:00
gaasedelen b82dba0d60 tweak: removes unnecessary composer shell border imposed by some Qt configs 2019-04-16 14:52:32 -04:00
gaasedelen e6f02ab873 allow smooth horizontal scrolling in xref window 2019-04-02 17:20:57 -04:00
gaasedelen 8316012782 switch xref attribution to bb granularity 2019-04-02 17:03:15 -04:00
gaasedelen fa3a13a085 allow for xrefing a batch 2019-04-02 12:26:42 -04:00
gaasedelen d6d0fbc7dd allow for smooth horizontal scrolling in coverage table 2019-04-01 13:54:42 -04:00
gaasedelen 1463949921 Merge branch 'fbl_block_blame' into develop, resolves #8 2019-03-31 19:56:36 -04:00
gaasedelen b6bf203b8c cleanup IDA integration of xrefs 2019-03-31 19:48:30 -04:00
gaasedelen 8052798c23 cleanup the xref dialog a bit 2019-03-31 19:17:47 -04:00
gaasedelen f89f3609f9 fix issue where coverage names are not saved to composed sets 2019-03-31 19:06:57 -04:00
gaasedelen 02d52fce73 add timestamps to coverage 2019-03-31 18:55:24 -04:00
gaasedelen 44cb1c8113 rough coverage xref dialog 2019-03-31 17:57:57 -04:00
gaasedelen dd50a3a8c8 rough draft of coverage xref for ida #8 2019-03-30 17:52:23 -04:00
gaasedelen ffc45f86a8 further refactor some loading code 2019-03-30 14:13:50 -04:00
gaasedelen e4ea2956e8 improve batch loading, refactor loading in general 2019-03-30 14:13:49 -04:00
gaasedelen 5468bef842 refactoring some loading code, add 'trace' format 2019-03-30 14:12:56 -04:00
gaasedelen 420c735600 rename testcases 2019-03-30 14:11:17 -04:00
gaasedelen beb715cf9b tweaks to reader & cov file code 2019-03-30 14:11:17 -04:00
gaasedelen 72fe0f90e6 add instruction sizes to metadata cache 2019-03-30 14:10:11 -04:00
xarkes 7cab1c1db9 Added python3 support (#62)
* Added Python3 support
* remove dependency of six library
* fixups based on Binja with Py3.6
2019-03-17 20:56:28 -04:00
gaasedelen b27698520b split metadata cache out of director 2019-03-15 19:48:57 -04:00
gaasedelen a55ede77f9 bugfix: deleted functions were not properly removed from metadata cache on refresh 2019-03-15 19:47:26 -04:00
gaasedelen 0f51554d8c suppress some IDA warning messages during func renames 2019-03-15 19:43:26 -04:00
gaasedelen 50cfd522f5 fix bad code in recent coverage loading work 2019-03-15 19:42:17 -04:00
gaasedelen 03717b03a2 rough plumbing for additional coverage formats #41 2019-03-10 16:06:10 -04:00
gaasedelen 92dfae662c Merge pull request #53 from v-p-b/binja_api_fix
Binary Ninja API fixups
2019-02-15 15:41:23 -05:00
gaasedelen 3b1c776455 Merge pull request #56 from AndrewFasano/master
Add python3 support for drcov parser
2019-01-31 12:37:04 -05:00
Andrew Fasano 5423bbf7e9 Add python3 support for drcov parser
And remain compatible with python2
2019-01-25 15:50:24 -05:00
b 0a5870a65a Added comments 2019-01-06 09:36:18 +01:00
b f3fd77e863 Stable+Dev compatibility 2019-01-06 08:55:50 +01:00
b 4b5ceabed1 Binja API: Getters -> Data members 2019-01-02 16:06:27 +01:00
gaasedelen 675cc8738d Merge branch 'develop' 2018-12-12 13:49:09 -05:00
gaasedelen 1ab2212115 Merge pull request #51 from yrp604/patch-1
Fix buggy drcov parsers
2018-12-12 13:37:59 -05:00
gaasedelen 713ac9d2aa Merge branch 'develop' into patch-1 2018-12-12 13:37:34 -05:00
_yrp 357bc5f839 Fix buggy drcov parsers
These two are definitely wrong, the other one might be too.

Found when some jerk compiled their own linux drcov plugin -- otherwise the precompiled binaries still spit out v2 module tables.
2018-12-11 20:45:35 -08:00
gaasedelen 8d60f778ef fixes botched fix, #50 2018-12-09 21:37:03 -05:00
gaasedelen 677723ed18 group lighthouse plugin menus in binja 2018-12-02 07:54:45 -05:00
gaasedelen b6e9ffe576 update to work with binja dev (breaks stable :-x) 2018-12-02 07:49:22 -05:00
gaasedelen dae71152e4 fixes thread safety issue in IDA 7.2, #50 2018-12-02 07:35:07 -05:00
gaasedelen f071dc6dce minor updates to pin documentation, #46 2018-11-18 20:59:21 -05:00
gaasedelen 83a2d71402 Merge pull request #49 from Ayrx/revert-47-improve-frida-script
Revert "Improve Frida coverage collection on quick exits."
2018-11-06 14:27:58 -05:00
Terry Chia e890ac60e9 Revert "Improve Frida coverage collection on quick exits." 2018-11-06 16:04:19 +08:00
gaasedelen 2b82bb69d5 fix edge case that could crash painter thread 2018-11-06 01:00:22 -05:00
gaasedelen 7cc235bde4 fix pintool on newer versions of gcc #46 2018-11-05 20:34:29 -05:00
gaasedelen 53e791b96e Merge pull request #47 from Ayrx/improve-frida-script
Improve Frida coverage collection on quick exits.
2018-10-31 14:36:21 -04:00
Terry Chia 48fd65de34 Improve Frida coverage collection on quick exits. 2018-10-30 14:48:20 +08:00
gaasedelen e6cf1d8292 Update drcov format v4 parsing for Windows, #45 2018-10-19 15:05:37 -04:00
gaasedelen 04225a0351 avoid division by zero, fixes #44 2018-10-19 15:00:45 -04:00
gaasedelen d3ba2c0ad7 hotfix for binja versioning bug #42 2018-10-12 11:27:20 -04:00
gaasedelen a81aa6d590 hotfix for personal binja licenses 2018-10-11 10:39:01 -04:00
gaasedelen fbbdd91576 Merge branch 'hotfix-0.8.1' 2018-10-03 01:45:19 -04:00
gaasedelen 03cc67e629 further tweak table row height based on DPI 2018-10-03 01:44:41 -04:00
gaasedelen 116f82e6bc only disable painting (on clear) if other coverage is open 2018-10-02 16:45:42 -04:00
gaasedelen 3f22eddc82 auto-cache metadata when opening coverage overview 2018-10-02 16:44:15 -04:00
gaasedelen 2bc6f109db fix for minor regression 2018-10-02 16:00:25 -04:00
gaasedelen 316e7622d7 bumps version number 2018-09-29 17:40:44 -04:00
gaasedelen bf66b02df9 fixes DPI regressions 2018-09-29 17:39:24 -04:00
gaasedelen bd8862923e fixes regression that could cause IDA 6.8 --> 7.0 to hang on exit 2018-09-29 17:34:43 -04:00
95 changed files with 24000 additions and 4133 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
MIT License
Copyright (c) 2017-2018 Markus Gaasedelen
Copyright (c) 2017-2024 Markus Gaasedelen
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
+80 -102
View File
@@ -1,18 +1,20 @@
# Lighthouse - A Code Coverage Explorer for Reverse Engineers
# Lighthouse - A Coverage Explorer for Reverse Engineers
<p align="center">
<img alt="Lighthouse Plugin" src="screenshots/overview.gif"/>
</p>
## Overview
Lighthouse is a code coverage plugin for [IDA Pro](https://www.hex-rays.com/products/ida/), and [Binary Ninja](https://binary.ninja/). The plugin makes use of interactive disassemblers to map, explore, and visualize externally collected code coverage data when symbols or source may not be available for a given binary.
Lighthouse is a powerful code coverage explorer for [IDA Pro](https://www.hex-rays.com/products/ida/) and [Binary Ninja](https://binary.ninja/), providing software researchers with uniquely interactive controls to study execution maps for native applications without requiring symbols or source.
This plugin is labeled only as a prototype & code resource for the community.
This project placed 2nd in IDA's [2017 Plug-In Contest](https://hex-rays.com/contests_details/contest2017/) and was later [nominated](https://pwnies.com/lighthouse/) in the 2021 Pwnie Awards for its contributions to the security research industry.
Special thanks to [@0vercl0k](https://twitter.com/0vercl0k) for the inspiration.
## Releases
* v0.9 -- Python 3 support, custom coverage formats, coverage cross-refs, theming subsystem, much more.
* v0.8 -- Binary Ninja support, HTML coverage reports, consistent styling, many tweaks, bugfixes.
* v0.7 -- Frida, C++ demangling, context menu, function prefixing, tweaks, bugfixes.
* v0.6 -- Intel pintool, cyclomatic complexity, batch load, bugfixes.
@@ -22,96 +24,93 @@ Special thanks to [@0vercl0k](https://twitter.com/0vercl0k) for the inspiration.
* v0.2 -- Multifile support, performance improvements, bugfixes.
* v0.1 -- Initial release
# IDA Pro Installation
# Installation
Lighthouse is a cross-platform (Windows, macOS, Linux) python plugin, supporting IDA Pro 6.8 and newer.
Lighthouse is a cross-platform (Windows, macOS, Linux) Python 2/3 plugin. It takes zero third party dependencies, making the code both portable and easy to install.
- Copy the contents of the `plugin` folder to the IDA plugins folder
- On Windows, the folder is at `C:\Program Files (x86)\IDA 6.8\plugins`
- On macOS, the folder is at `/Applications/IDA\ Pro\ 6.8/idaq.app/Contents/MacOS/plugins`
- On Linux, the folder may be at `/opt/IDA/plugins/`
Use the instructions below for your respective disassembler.
It has been primarily developed and tested on Windows, so that is where we expect the best experience.
## IDA Installation
# Binary Ninja Installation (Experimental)
1. From IDA's Python console, run the following command to find its plugin directory:
- `import idaapi, os; print(os.path.join(idaapi.get_user_idadir(), "plugins"))`
2. Copy the contents of this repository's `/plugins/` folder to the listed directory.
3. Restart your disassembler.
At this time, support for Binary Ninja is considered experimental. Please feel free to report any bugs that you encounter.
## Binary Ninja Installation
You can install Lighthouse & PyQt5 for Binary Ninja by following the instructions below.
Lighthouse can be installed through the plugin manager on Binary Ninja, supporting v3.5 and newer.
## Windows Installation
1. Install PyQt5 from a Windows command prompt with the following command:
```
pip install --target="%appdata%\Binary Ninja\plugins\Lib\site-packages" python-qt5
```
2. Copy the contents of the `/plugin/` folder in this repo to your Binary Ninja [plugins folder](https://docs.binary.ninja/guide/plugins/index.html#using-plugins).
## Linux Installation
1. Install PyQt5 from a Linux shell with the following command:
```
sudo apt install python-pyqt5
```
2. Copy the contents of the `/plugin/` folder in this repo to your Binary Ninja [plugins folder](https://docs.binary.ninja/guide/plugins/index.html#using-plugins).
## macOS Installation
¯\\\_(ツ)\_/¯
1. Open Binary Ninja's plugin manager by navigating the following submenus:
- `Edit` -> `Preferences` -> `Manage Plugins`
2. Search for Lighthouse in the plugin manager, and click the `Enable` button in the bottom right.
3. Restart your disassembler.
# Usage
Lighthouse loads automatically when a database is opened, installing a handful of menu entries into the disassembler.
Once properly installed, there will be a few new menu entries available in the disassembler. These are the entry points for a user to load coverage data and start using Lighthouse.
<p align="center">
<img alt="Lighthouse Menu Entries" src="screenshots/open.gif"/>
</p>
These are the entry points for a user to load and view coverage data.
Lighthouse is able to load a few different 'flavors' of coverage data. To generate coverage data that can be loaded into Lighthouse, please look at the [README](https://github.com/gaasedelen/lighthouse/tree/master/coverage) in the coverage directory of this repository.
## Coverage Painting
Lighthouse 'paints' the active coverage data across the three major IDA views as applicable. Specifically, the Disassembly, Graph, and Pseudocode views.
While Lighthouse is in use, it will 'paint' the active coverage data across all of the code viewers available in the disassembler. Specifically, this will apply to your linear disassembly, graph, and decompiler windows.
<p align="center">
<img alt="Lighthouse Coverage Painting" src="screenshots/painting.png"/>
</p>
In Binary Ninja, only the Disassembly and Graph views are supported.
In Binary Ninja, only the linear disassembly, graph, and IL views are supported. Support for painting decompiler output in Binary Ninja will be added to Lighthouse in the *near future* as the feature stabilizes.
## Coverage Overview
# Coverage Overview
The Coverage Overview is a dockable widget that provides a function level view of the active coverage data for the database.
The Coverage Overview is a dockable widget that will open up once coverage has been loaded into Lighthouse.
<p align="center">
<img alt="Lighthouse Coverage Overview" src="screenshots/overview.png"/>
</p>
This table can be sorted by column, and entries can be double clicked to jump to their corresponding disassembly.
This interactive widget provides a function level view of the loaded coverage data. It also houses a number of tools to manage loaded data and drive more advanced forms of coverage analysis.
## Context Menu
Right clicking the table in the Coverage Overview will produce a context menu with a few basic amenities.
Right clicking the table in the Coverage Overview will produce a context menu with a few basic amenities to extract information from the table, or manipulate the database as part of your reverse engineering process.
<p align="center">
<img alt="Lighthouse Context Menu" src="screenshots/context_menu.gif"/>
</p>
These actions can be used to quickly manipulate or interact with entries in the table.
If there are any other actions that you think might be useful to add to this context menu, please file an issue and they will be considered for a future release of Lighthouse.
## Coverage Composition
## Coverage ComboBox
Building relationships between multiple sets of coverage data often distills deeper meaning than their individual parts. The shell at the bottom of the [Coverage Overview](#coverage-overview) provides an interactive means of constructing these relationships.
Loaded coverage and user constructed compositions can be selected or deleted through the coverage combobox.
<p align="center">
<img alt="Lighthouse Coverage ComboBox" src="screenshots/combobox.gif"/>
</p>
## HTML Coverage Report
Lighthouse can generate rudimentary HTML coverage reports. A sample report can be seen [here](https://rawgit.com/gaasedelen/lighthouse/master/testcase/report.html).
<p align="center">
<img alt="Lighthouse HTML Report" src="screenshots/html_report.gif"/>
</p>
# Coverage Shell
At the bottom of the coverage overview window is the coverage shell. This shell can be used to perform logic-based operations that combine or manipulate the loaded coverage sets.
<p align="center">
<img alt="Lighthouse Coverage Composition" src="screenshots/shell.gif"/>
</p>
Pressing `enter` on the shell will evaluate and save a user constructed composition.
This feature is extremely useful in exploring the relationships of program execution across multiple runs. In other words, the shell can be used to 'diff' execution between coverage sets and extract a deeper meaning that is otherwise obscured within the noise of their individual parts.
## Composition Syntax
@@ -119,16 +118,27 @@ Coverage composition, or _Composing_ as demonstrated above is achieved through a
### Grammar Tokens
* Logical Operators: `|, &, ^, -`
* Coverage Symbol: `A, B, C, ..., Z`
* Coverage Range: `A,C`, `Q,Z`, ...
* Coverage Symbol: `A, B, C, ..., Z, *`
* Parenthesis: `(...)`
### Example Compositions
* `A & B`
* `(A & B) | C`
* `(C & (A - B)) | (F,H & Q)`
The evaluation of the composition may occur right to left, parenthesis are suggested for potentially ambiguous expressions.
1. Executed code that is *shared* between coverage `A` and coverage `B`:
```
A & B
```
2. Executed code that is *unique* only to coverage `A`:
```
A - B
```
3. Executed code that is *unique* to `A` or `B`, but not `C`:
```
(A | B) - C
```
Expressions can be of arbitrary length or complexity, but the evaluation of the composition may occur right to left. So parenthesis are suggested for potentially ambiguous expressions.
## Hot Shell
@@ -142,7 +152,7 @@ The hot shell serves as a natural gateway into the unguided exploration of compo
## Search
Using the shell, one can search and filter the functions listed in the coverage table by prefixing their query with `/`.
Using the shell, you can search and filter the functions listed in the coverage table by prefixing their query with `/`.
<p align="center">
<img alt="Lighthouse Search" src="screenshots/search.gif"/>
@@ -158,76 +168,44 @@ Entering an address or function name into the shell can be used to jump to corre
<img alt="Lighthouse Jump" src="screenshots/jump.gif"/>
</p>
## Coverage ComboBox
# Coverage Cross-references (Xref)
Loaded coverage data and user constructed compositions can be selected or deleted through the coverage combobox.
While using Lighthouse, you can right click any basic block (or instruction) and use the 'Coverage Xref' action to see which coverage sets executed the selected block. Double clicking any of the listed entries will instantly switch to that coverage set.
<p align="center">
<img alt="Lighthouse Coverage ComboBox" src="screenshots/combobox.gif"/>
<img alt="Lighthouse Xref" src="screenshots/xref.gif"/>
</p>
## HTML Coverage Report
This pairs well with the 'Coverage Batch' feature, which allows you to quickly load and aggregate thousands of coverage files into Lighthouse. Cross-referencing a block and selecting a 'set' will load the 'guilty' set from disk as a new coverage set for you to explore separate from the batch.
Lighthouse can generate a rudimentary HTML coverage report of the active coverage.
A sample report can be seen [here](https://rawgit.com/gaasedelen/lighthouse/master/testcase/report.html).
# Themes
Lighthouse ships with two default themes -- a 'light' theme, and a 'dark' one. Depending on the colors currently used by your disassembler, Lighthouse will attempt to select the theme that seems most appropriate.
<p align="center">
<img alt="Lighthouse HTML Report" src="screenshots/html_report.gif"/>
<img alt="Lighthouse Themes" src="screenshots/themes.png"/>
</p>
# Collecting Coverage
The theme files are stored as simple JSON on disk and are highly configurable. If you are not happy with the default themes or colors, you can create your own themes and simply drop them in the user theme directory.
Before using Lighthouse, one will need to collect code coverage data for their target binary / application.
The examples below demonstrate how one can use [DynamoRIO](http://www.dynamorio.org), [Intel Pin](https://software.intel.com/en-us/articles/pin-a-dynamic-binary-instrumentation-tool) or [Frida](https://www.frida.re) to collect Lighthouse compatible coverage against a target. The `.log` files produced by these instrumentation tools can be loaded directly into Lighthouse.
## DynamoRIO
Code coverage data can be collected via DynamoRIO's [drcov](http://dynamorio.org/docs/page_drcov.html) code coverage module.
Example usage:
```
..\DynamoRIO-Windows-7.0.0-RC1\bin64\drrun.exe -t drcov -- boombox.exe
```
## Intel Pin
Using a [custom pintool](coverage/pin) contributed by [Agustin Gianni](https://twitter.com/agustingianni), the Intel Pin DBI can also be used to collect coverage data.
Example usage:
```
pin.exe -t CodeCoverage64.dll -- boombox.exe
```
For convenience, binaries for the Windows pintool can be found on the [releases](https://github.com/gaasedelen/lighthouse/releases/tag/v0.8.0) page. macOS and Linux users need to compile the pintool themselves following the [instructions](coverage/pin#compilation) included with the pintool for their respective platforms.
## Frida (Experimental)
Lighthouse offers limited support for Frida based code coverage via a custom [instrumentation script](coverage/frida) contributed by [yrp](https://twitter.com/yrp604).
Example usage:
```
sudo python frida-drcov.py bb-bench
```
Lighthouse will remember your theme preference for future loads and uses.
# Future Work
Time and motivation permitting, future work may include:
* Nag Vector35 to fix HLIL highlighting ([bug](https://github.com/Vector35/binaryninja-api/issues/2584)) in Binary Ninja
* ~~Asynchronous composition, painting, metadata collection~~
* ~~Multifile/coverage support~~
* Profiling based heatmaps/painting
* Coverage & profiling treemaps
* ~~Additional coverage sources, trace formats, etc~~
* Improved pseudocode painting
* Lighthouse console access, headless usage
* Custom themes
* Python 3 support
* ~~Lighthouse console access~~, headless usage
* ~~Custom themes~~
* ~~Python 3 support~~
I welcome external contributions, issues, and feature requests.
I welcome external contributions, issues, and feature requests. Please make any pull requests to the `develop` branch of this repository if you would like them to be considered for a future release.
# Authors
+11
View File
@@ -0,0 +1,11 @@
# Lighthouse - A Coverage Explorer for Reverse Engineers
<p align="center">
<img alt="Lighthouse Plugin" src="https://raw.githubusercontent.com/gaasedelen/lighthouse/master/screenshots/overview.gif"/>
</p>
## Overview
Lighthouse is a powerful code coverage explorer for [IDA Pro](https://www.hex-rays.com/products/ida/) and [Binary Ninja](https://binary.ninja/), providing software researchers with uniquely interactive controls to study execution maps for native applications without requiring symbols or source.
For additional usage information, please check out the full [README](https://github.com/gaasedelen/lighthouse) on GitHub.
+26
View File
@@ -0,0 +1,26 @@
import os
import sys
#------------------------------------------------------------------------------
# Binary Ninja 'Plugin Manager' Stub
#------------------------------------------------------------------------------
#
# This file is an alternative loading stub created specifically to
# support the ability to 'easy' install Lighthouse into Binary Ninja
# via its 'Plugin Manager' functionality.
#
# Please disregard this code / subdirectory if performing **manual**
# installations of Lighthouse in IDA or Binary Ninja.
#
lh_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "..", "plugins")
sys.path.append(lh_path)
from lighthouse.util.log import logging_started, start_logging
from lighthouse.util.disassembler import disassembler
if not logging_started():
logger = start_logging()
logger.info("Selecting Binary Ninja loader...")
from lighthouse.integration.binja_loader import *
+24
View File
@@ -0,0 +1,24 @@
{
"api": [
"python3"
],
"author": "Markus Gaasedelen",
"description": "A Coverage Explorer for Reverse Engineers",
"license": {
"name": "MIT",
"text": "Copyright (c) 2024> Markus Gaasedelen\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
},
"longdescription": "",
"minimumbinaryninjaversion": 4526,
"name": "Lighthouse",
"platforms": [
"Darwin",
"Linux",
"Windows"
],
"pluginmetadataversion": 2,
"type": [
"helper"
],
"version": "0.9.4"
}
+79
View File
@@ -0,0 +1,79 @@
# Collecting Coverage
Before using Lighthouse, one will need to collect code coverage data for their target binary / application.
The examples below demonstrate how one can use [DynamoRIO](http://www.dynamorio.org), [Intel Pin](https://software.intel.com/en-us/articles/pin-a-dynamic-binary-instrumentation-tool) or [Frida](https://www.frida.re) to collect Lighthouse compatible coverage against a target. The `.log` files produced by these instrumentation tools can be loaded directly into Lighthouse.
## DynamoRIO
Code coverage data can be collected via DynamoRIO's [drcov](http://dynamorio.org/docs/page_drcov.html) code coverage module.
Example usage:
```
..\DynamoRIO-Windows-7.0.0-RC1\bin64\drrun.exe -t drcov -- boombox.exe
```
## Intel Pin
Using a [custom pintool](pin/README.md) contributed by [Agustin Gianni](https://twitter.com/agustingianni), the Intel Pin DBI can also be used to collect coverage data.
Example usage:
```
pin.exe -t CodeCoverage64.dll -- boombox.exe
```
For convenience, binaries for the Windows pintool can be found on the [releases](https://github.com/gaasedelen/lighthouse/releases) page. macOS and Linux users need to compile the pintool themselves following the [instructions](pin/README.md#compilation) included with the pintool for their respective platforms.
## Frida (Experimental)
Lighthouse offers limited support for Frida based code coverage via a custom [instrumentation script](frida/README.md) contributed by [yrp](https://twitter.com/yrp604).
Example usage:
```
sudo python frida-drcov.py bb-bench
```
# Other Coverage Formats
Lighthouse is flexible as to what kind of coverage or 'trace' file formats it can load. Below is an outline of these human-readable text formats that are arguably the easiest to output from a custom tracer.
## Module + Offset (modoff)
A 'Module+Offset' coverage file / trace is a highly recommended coverage format due to its simplicity and readability:
```
boombox+3a06
boombox+3a09
boombox+3a0f
boombox+3a15
...
```
Each line of the trace represents an executed instruction or basic block in the instrumented program. The line *must* name an executed module eg `boombox.exe` and a relative offset to the executed address from the imagebase.
It is okay for hits from other modules (say, `kernel32.dll`) to exist in the trace. Lighthouse will not load coverage for them.
## Address Trace (Instruction, or Basic Block)
Perhaps the most primitive coverage format, Lighthouse can also consume an 'absolute address' style trace:
```
0x14000419c
0x1400041a0
0x1400045dc
0x1400045e1
0x1400045e2
...
```
Note that these address traces can be either instruction addresses, or basic block addresses -- it does not matter. The main caveat is that addresses in the trace *must* match the address space within the disassembler database.
If an address cannot be mapped into a function in the disassembler database, Lighthouse will simply discard it.
## Custom Trace Formats
If you are adamant to use a completely custom coverage format, you can try to subclass Lighthouse's `CoverageFile` parser interface. Once complete, simply drop your parser into the `parsers` folder.
+5 -5
View File
@@ -65,7 +65,7 @@ maps.map(function (e) {
var filtered_maps = new ModuleMap(function (m) {
if (whitelist.indexOf('all') >= 0) { return true; }
return whitelist.indexOf(m.name) >= 0;
return whitelist.some(item => m.name.toLowerCase().includes(item.toLowerCase()));
});
// This function takes a list of GumCompileEvents and converts it into a DRcov
@@ -227,12 +227,12 @@ def create_header(mods):
header_modules = '\n'.join(entries)
return header + header_modules + '\n'
return ("%s%s\n" % (header, header_modules)).encode("utf-8")
# take the recv'd basic blocks, finish the header, and append the coverage
def create_coverage(data):
bb_header = 'BB Table: %d bbs\n' % len(data)
return bb_header + ''.join(data)
bb_header = b'BB Table: %d bbs\n' % len(data)
return bb_header + b''.join(data)
def on_message(msg, data):
#print(msg)
@@ -323,7 +323,7 @@ def main():
script.on('message', on_message)
script.load()
print('[*] Now collecting info, control-D to terminate....')
print('[*] Now collecting info, control-C or control-D to terminate....')
sys.stdin.read()
+38 -26
View File
@@ -25,32 +25,29 @@ using unordered_map = std::tr1::unordered_map<K, V>;
}
// Tool's arguments.
static KNOB<string> KnobModuleWhitelist(KNOB_MODE_APPEND, "pintool", "w", "",
static KNOB<std::string> KnobModuleWhitelist(KNOB_MODE_APPEND, "pintool", "w", "",
"Add a module to the white list. If none is specified, everymodule is white-listed. Example: libTIFF.dylib");
static KNOB<string> KnobLogFile(KNOB_MODE_WRITEONCE, "pintool", "l", "trace.log",
static KNOB<std::string> KnobLogFile(KNOB_MODE_WRITEONCE, "pintool", "l", "trace.log",
"Name of the output file. If none is specified, trace.log is used.");
// Return the file/directory name of a path.
static string base_name(const string& path)
static std::string base_name(const std::string& path)
{
#if defined(TARGET_WINDOWS)
#define PATH_SEPARATOR "\\"
#else
#define PATH_SEPARATOR "/"
#endif
string::size_type idx = path.rfind(PATH_SEPARATOR);
string name = (idx == string::npos) ? path : path.substr(idx + 1);
std::string::size_type idx = path.rfind(PATH_SEPARATOR);
std::string name = (idx == std::string::npos) ? path : path.substr(idx + 1);
return name;
}
// Per thread data structure. This is mainly done to avoid locking.
// - Per-thread map of executed basic blocks, and their size.
struct ThreadData {
// Unique list of hit basic blocks.
pintool::unordered_set<ADDRINT> m_block_hit;
// Map basic a block address to its size.
pintool::unordered_map<ADDRINT, uint16_t> m_block_size;
pintool::unordered_map<ADDRINT, uint16_t> m_blocks;
};
class ToolContext {
@@ -129,7 +126,7 @@ static VOID OnThreadFini(THREADID tid, const CONTEXT* ctxt, INT32 c, VOID* v)
static VOID OnImageLoad(IMG img, VOID* v)
{
auto& context = *reinterpret_cast<ToolContext*>(v);
string img_name = base_name(IMG_Name(img));
std::string img_name = base_name(IMG_Name(img));
ADDRINT low = IMG_LowAddress(img);
ADDRINT high = IMG_HighAddress(img);
@@ -165,24 +162,37 @@ static VOID PIN_FAST_ANALYSIS_CALL OnBasicBlockHit(THREADID tid, ADDRINT addr, U
{
auto& context = *reinterpret_cast<ToolContext*>(v);
ThreadData* data = context.GetThreadLocalData(tid);
data->m_block_hit.insert(addr);
data->m_block_size[addr] = size;
data->m_blocks[addr] = size;
PIN_RemoveInstrumentationInRange(addr, addr);
}
// Trace hit event handler.
static VOID OnTrace(TRACE trace, VOID* v)
{
auto& context = *reinterpret_cast<ToolContext*>(v);
BBL bbl = TRACE_BblHead(trace);
ADDRINT addr = BBL_Address(bbl);
// Check if the address is inside a white-listed image.
if (!context.m_tracing_enabled || !context.m_images->isInterestingAddress(addr))
if (!context.m_tracing_enabled || !context.m_images->isInterestingAddress(TRACE_Address(trace)))
return;
// For each basic block in the trace.
for (; BBL_Valid(bbl); bbl = BBL_Next(bbl)) {
addr = BBL_Address(bbl);
auto tid = PIN_ThreadId();
ThreadData* data = context.GetThreadLocalData(tid);
// This trace is getting JIT'd, which implies the head must get executed.
auto bbl = TRACE_BblHead(trace);
auto addr = BBL_Address(bbl);
data->m_blocks[addr] = (uint16_t)BBL_Size(bbl);
// For each basic block in the trace...
for (bbl = BBL_Next(bbl); BBL_Valid(bbl); bbl = BBL_Next(bbl))
{
// Ignore blocks that have already been marked as executed in the past...
ADDRINT addr = BBL_Address(bbl);
if (data->m_blocks.find(addr) != data->m_blocks.end())
continue;
// Instrument blocks that have not yet been executed (at least... by this thread).
BBL_InsertCall(bbl, IPOINT_ANYWHERE, (AFUNPTR)OnBasicBlockHit,
IARG_FAST_ANALYSIS_CALL,
IARG_THREAD_ID,
@@ -191,6 +201,7 @@ static VOID OnTrace(TRACE trace, VOID* v)
IARG_PTR, v,
IARG_END);
}
}
// Program finish event handler.
@@ -218,7 +229,7 @@ static VOID OnFini(INT32 code, VOID* v)
// Count the global number of basic blocks.
size_t number_of_bbs = 0;
for (const auto& data : context.m_terminated_threads) {
number_of_bbs += data->m_block_hit.size();
number_of_bbs += data->m_blocks.size();
}
context.m_trace->write_string("BB Table: %u bbs\n", number_of_bbs);
@@ -232,7 +243,8 @@ static VOID OnFini(INT32 code, VOID* v)
drcov_bb tmp;
for (const auto& data : context.m_terminated_threads) {
for (const auto& address : data->m_block_hit) {
for (const auto& block : data->m_blocks) {
auto address = block.first;
auto it = std::find_if(context.m_loaded_images.begin(), context.m_loaded_images.end(), [&address](const LoadedImage& image) {
return address >= image.low_ && address < image.high_;
});
@@ -242,7 +254,7 @@ static VOID OnFini(INT32 code, VOID* v)
tmp.id = (uint16_t)std::distance(context.m_loaded_images.begin(), it);
tmp.start = (uint32_t)(address - it->low_);
tmp.size = data->m_block_size[address];
tmp.size = data->m_blocks[address];
context.m_trace->write_binary(&tmp, sizeof(tmp));
}
@@ -251,14 +263,14 @@ static VOID OnFini(INT32 code, VOID* v)
int main(int argc, char* argv[])
{
cout << "CodeCoverage tool by Agustin Gianni (agustingianni@gmail.com)" << endl;
std::cout << "CodeCoverage tool by Agustin Gianni (agustingianni@gmail.com)" << std::endl;
// Initialize symbol processing
PIN_InitSymbols();
// Initialize PIN.
if (PIN_Init(argc, argv)) {
cerr << "Error initializing PIN, PIN_Init failed!" << endl;
std::cerr << "Error initializing PIN, PIN_Init failed!" << std::endl;
return -1;
}
@@ -268,7 +280,7 @@ int main(int argc, char* argv[])
// Create a an image manager that keeps track of the loaded/unloaded images.
context->m_images = new ImageManager();
for (unsigned i = 0; i < KnobModuleWhitelist.NumberOfValues(); ++i) {
cout << "White-listing image: " << KnobModuleWhitelist.Value(i) << endl;
std::cout << "White-listing image: " << KnobModuleWhitelist.Value(i) << std::endl;
context->m_images->addWhiteListedImage(KnobModuleWhitelist.Value(i));
// We will only enable tracing when any of the whitelisted images gets loaded.
@@ -276,7 +288,7 @@ int main(int argc, char* argv[])
}
// Create a trace file.
cout << "Logging code coverage information to: " << KnobLogFile.ValueString() << endl;
std::cout << "Logging code coverage information to: " << KnobLogFile.ValueString() << std::endl;
context->m_trace = new TraceFile(KnobLogFile.ValueString());
// Handlers for thread creation and destruction.
+2 -2
View File
@@ -11,7 +11,7 @@ ImageManager::~ImageManager()
PIN_RWMutexFini(&images_lock);
}
VOID ImageManager::addImage(string image_name, ADDRINT lo_addr,
VOID ImageManager::addImage(std::string image_name, ADDRINT lo_addr,
ADDRINT hi_addr)
{
PIN_RWMutexWriteLock(&images_lock);
@@ -25,7 +25,7 @@ VOID ImageManager::removeImage(ADDRINT low)
{
PIN_RWMutexWriteLock(&images_lock);
{
set<LoadedImage>::iterator i = images.find(LoadedImage("", low));
std::set<LoadedImage>::iterator i = images.find(LoadedImage("", low));
if (i != images.end()) {
LoadedImage li = *i;
images.erase(i);
+1 -1
View File
@@ -7,4 +7,4 @@ TOOL_ROOTS := CodeCoverage
$(OBJDIR)CodeCoverage$(PINTOOL_SUFFIX): $(OBJDIR)CodeCoverage$(OBJ_SUFFIX) $(OBJDIR)ImageManager$(OBJ_SUFFIX)
$(LINKER) $(TOOL_LDFLAGS) $(LINK_EXE)$@ $^ $(TOOL_LPATHS) $(TOOL_LIBS)
include $(TOOLS_ROOT)/Config/makefile.default.rules
include $(TOOLS_ROOT)/Config/makefile.default.rules
+15 -6
View File
@@ -1,6 +1,6 @@
# CodeCoverage Pintool
The `CodeCoverage` pintool runs ontop of the [Intel Pin](https://software.intel.com/en-us/articles/pin-a-dynamic-binary-instrumentation-tool) DBI framework and collects code coverage data in a log format compatible with [Lighthouse](https://github.com/gaasedelen/lighthouse). The log produced by this pintool emulates that of [drcov](http://dynamorio.org/docs/page_drcov.html) as shipped with [DynamoRIO](http://www.dynamorio.org).
The `CodeCoverage` pintool runs ontop of the [Intel Pin](https://software.intel.com/en-us/articles/pin-a-dynamic-binary-instrumentation-tool) DBI framework and collects code coverage data in a log format compatible with [Lighthouse](https://github.com/gaasedelen/lighthouse). The log produced by this pintool emulates that of [drcov](http://dynamorio.org/docs/page_drcov.html) as shipped with [DynamoRIO](http://www.dynamorio.org).
This pintool is labeled only as a prototype.
@@ -12,7 +12,7 @@ Follow the build instructions below for your respective platform.
## Building for MacOS or Linux
On MacOS or Liunux, one can compile the pintool using the following commands.
On MacOS or Linux, one can compile the pintool using the following commands.
```
# Location of this repo / pintool source
@@ -22,6 +22,7 @@ cd ~/lighthouse/coverage/pin
export PIN_ROOT=~/pin
export PATH=$PATH:$PIN_ROOT
make
make TARGET=ia32
```
The resulting binaries will be placed inside a directory whose name depends on the arch/platform/build type.
@@ -38,7 +39,11 @@ Launch a command prompt and build the pintool with the following commands.
### 32bit Pintool
```
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x86
REM If you are on VS 2022 or so you can run this line:
"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" x86
REM VS 2015 or so you can run this line instead:
REM "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x86
REM Location of this repo / pintool source
cd C:\Users\user\lighthouse\coverage\pin
@@ -52,7 +57,11 @@ build-x86.bat
### 64bit Pintool
```
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x86_amd64
REM If you are on VS 2022 or so you can run this line:
"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" x86_amd64
REM VS 2015 or so you can run this line instead:
REM "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x86_amd64
REM Location of this repo / pintool source
cd C:\Users\user\lighthouse\coverage\pin
@@ -63,12 +72,12 @@ set PATH=%PATH%;%PIN_ROOT%
build-x64.bat
```
The resulting binaries will be labaled based on their architecture (eg, 64 is the 64bit pintool).
The resulting binaries will be labeled based on their architecture (eg, 64 is the 64bit pintool).
* CodeCoverage.dll
* CodeCoverage64.dll
Compiling a pintool on Windows can be more arduous. Because of this, we have provided compiled binaries for Windows on the [releases](https://github.com/gaasedelen/lighthouse/releases/tag/v0.7.0) page.
Compiling a pintool on Windows can be more arduous. Because of this, we have provided compiled binaries for Windows on the [releases](https://github.com/gaasedelen/lighthouse/releases) page. Please be sure to use the pintool that matches your version of Pin.
# Usage
+6 -7
View File
@@ -2,11 +2,12 @@
cls
cl ^
/c ^
/c /Fo /nologo /EHa- /EHs- /GR- /GS- /Gd /Gm- /Gy /MD /O2 /Oi- /Oy- /TP /W3 /WX- /Zc:forScope /Zc:inline /Zc:wchar_t /wd4316 /wd4530 /fp:strict ^
/DTARGET_IA32E /DHOST_IA32E /DTARGET_WINDOWS /DWIN32 /D__PIN__=1 /DPIN_CRT=1 /D_STLP_IMPORT_IOSTREAMS /D__LP64__ ^
/I"%PIN_ROOT%\extras\xed-intel64\include\xed" ^
/I%PIN_ROOT%\source\include\pin ^
/I%PIN_ROOT%\source\include\pin\gen ^
/I%PIN_ROOT%\source\tools\InstLib ^
/I"%PIN_ROOT%\extras\xed-intel64\include\xed" ^
/I%PIN_ROOT%\extras\components\include ^
/I%PIN_ROOT%\extras\stlport\include ^
/I%PIN_ROOT%\extras ^
@@ -16,9 +17,6 @@ cl ^
/I"%PIN_ROOT%\extras\crt\include\arch-x86_64" ^
/I%PIN_ROOT%\extras\crt\include\kernel\uapi ^
/I"%PIN_ROOT%\extras\crt\include\kernel\uapi\asm-x86" ^
/nologo /W3 /WX- /O2 ^
/D TARGET_IA32E /D HOST_IA32E /D TARGET_WINDOWS /D WIN32 /D __PIN__=1 /D PIN_CRT=1 /D __LP64__ ^
/Gm- /MT /GS- /Gy /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /GR- /Gd /TP /wd4530 /GR- /GS- /EHs- /EHa- /FP:strict /Oi- ^
/FIinclude/msvc_compat.h CodeCoverage.cpp ImageManager.cpp ImageManager.h TraceFile.h
link ^
@@ -29,7 +27,7 @@ link ^
/LIBPATH:%PIN_ROOT%\intel64\lib ^
/LIBPATH:"%PIN_ROOT%\intel64\lib-ext" ^
/LIBPATH:"%PIN_ROOT%\extras\xed-intel64\lib" ^
/LIBPATH:%PIN_ROOT%\intel64\runtime\pincrt pin.lib xed.lib pinvm.lib kernel32.lib "stlport-static.lib" "m-static.lib" "c-static.lib" "os-apis.lib" "ntdll-64.lib" crtbeginS.obj ^
/LIBPATH:%PIN_ROOT%\intel64\runtime\pincrt pin.lib xed.lib pinipc.lib pincrt.lib kernel32.lib crtbeginS.obj ^
/NODEFAULTLIB ^
/MANIFEST:NO ^
/OPT:NOREF ^
@@ -45,6 +43,7 @@ link ^
/ignore:4049 ^
/ignore:4210 ^
/ignore:4217 ^
/ignore:4281 ^
/DLL CodeCoverage.obj ImageManager.obj
del *.obj *.pdb *.exp *.lib
del *.obj *.pdb *.exp *.lib
+4 -3
View File
@@ -2,8 +2,8 @@
cls
cl ^
/c /EHa- /EHs- /GR- /GS- /Gd /Gm- /Gy /MT /O2 /Oi- /Oy- /TP /W3 /WX- /Zc:forScope /Zc:inline /Zc:wchar_t /fp:precise /nologo /wd4316 ^
/DTARGET_IA32 /DHOST_IA32 /DTARGET_WINDOWS /DBIGARRAY_MULTIPLIER=1 /DWIN32 /D__PIN__=1 /DPIN_CRT=1 /D__i386__ ^
/c /Fo /nologo /EHa- /EHs- /GR- /GS- /Gd /Gm- /Gy /MD /O2 /Oi- /Oy- /TP /W3 /WX- /Zc:forScope /Zc:inline /Zc:wchar_t /wd4316 /wd4530 /fp:precise ^
/DTARGET_IA32 /DHOST_IA32 /DTARGET_WINDOWS /DWIN32 /D__PIN__=1 /DPIN_CRT=1 /D_STLP_IMPORT_IOSTREAMS /D__i386__ ^
/I"%PIN_ROOT%\extras\xed-ia32\include\xed" ^
/I%PIN_ROOT%\source\include\pin ^
/I%PIN_ROOT%\source\include\pin\gen ^
@@ -28,7 +28,7 @@ link ^
/LIBPATH:%PIN_ROOT%\ia32\lib ^
/LIBPATH:"%PIN_ROOT%\ia32\lib-ext" ^
/LIBPATH:"%PIN_ROOT%\extras\xed-ia32\lib" ^
/LIBPATH:%PIN_ROOT%\ia32\runtime\pincrt pin.lib xed.lib pinvm.lib kernel32.lib "stlport-static.lib" "m-static.lib" "c-static.lib" "os-apis.lib" "ntdll-32.lib" crtbeginS.obj ^
/LIBPATH:%PIN_ROOT%\ia32\runtime\pincrt pin.lib xed.lib pinipc.lib pincrt.lib kernel32.lib crtbeginS.obj ^
/NODEFAULTLIB ^
/MANIFEST:NO ^
/OPT:NOREF ^
@@ -44,6 +44,7 @@ link ^
/ignore:4049 ^
/ignore:4210 ^
/ignore:4217 ^
/ignore:4281 ^
/DLL CodeCoverage.obj ImageManager.obj
del *.obj *.pdb *.exp *.lib
+1 -1
View File
@@ -14,5 +14,5 @@ xcopy /s/y "..\plugin\*" "C:\Users\user\AppData\Roaming\Binary Ninja\plugins\"
del /F /Q "C:\Users\user\AppData\Roaming\Binary Ninja\plugins\.#lighthouse_plugin.py"
REM - Launch a new IDA session
start "" "C:\tools\disassemblers\BinaryNinja\binaryninja.exe" "..\..\testcase\idaq.bndb"
start "" "C:\tools\disassemblers\BinaryNinja\binaryninja.exe" "..\..\testcase\ida74\ida64.bndb"
+18
View File
@@ -0,0 +1,18 @@
set LIGHTHOUSE_LOGGING=1
REM - Close any running instances of IDA
call close_BINJA.bat
REM - Purge old lighthouse log files
del /F /Q "C:\Users\user\AppData\Roaming\Binary Ninja\lighthouse_logs\*"
REM - Delete the old plugin bits
del /F /Q "C:\Users\user\AppData\Roaming\Binary Ninja\plugins\*lighthouse_plugin.py"
rmdir "C:\Users\user\AppData\Roaming\Binary Ninja\plugins\lighthouse" /s /q
REM - Copy over the new plugin bits
xcopy /s/y "..\plugin\*" "C:\Users\user\AppData\Roaming\Binary Ninja\plugins\"
del /F /Q "C:\Users\user\AppData\Roaming\Binary Ninja\plugins\.#lighthouse_plugin.py"
REM - Launch a new IDA session
start "" "C:\tools\disassemblers\BinaryNinja\binaryninja.exe"
+18
View File
@@ -0,0 +1,18 @@
set LIGHTHOUSE_LOGGING=1
REM - Close any running instances of Binja
call close_BINJA.bat
REM - Purge old lighthouse log files
del /F /Q "C:\Users\user\AppData\Roaming\Binary Ninja\lighthouse_logs\*"
REM - Delete the old plugin bits
del /F /Q "C:\Users\user\AppData\Roaming\Binary Ninja\plugins\*lighthouse_plugin.py"
rmdir "C:\Users\user\AppData\Roaming\Binary Ninja\plugins\lighthouse" /s /q
REM - Copy over the new plugin bits
xcopy /s/y "..\plugin\*" "C:\Users\user\AppData\Roaming\Binary Ninja\plugins\"
del /F /Q "C:\Users\user\AppData\Roaming\Binary Ninja\plugins\.#lighthouse_plugin.py"
REM - Launch a new Binja session
start "" "C:\tools\disassemblers\BinaryNinja_Personal\binaryninja.exe" "..\..\testcase\boombox.bndb"
@@ -13,6 +13,6 @@ REM - Copy over the new plugin bits
xcopy /s/y "..\plugin\*" "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\plugins\"
del /F /Q "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\plugins\.#lighthouse_plugin.py"
REM - Relaunch two IDA sessions
start "" "C:\tools\disassemblers\IDA 6.95\idaq64.exe" "..\..\testcase\boombox95.i64"
REM - Launch a new IDA session
start "" "C:\tools\disassemblers\IDA 7.4\ida64.exe" "..\..\testcase\boombox74.i64"
+19
View File
@@ -0,0 +1,19 @@
set LIGHTHOUSE_LOGGING=1
REM - Close any running instances of IDA
call close_IDA.bat
REM - Purge old lighthouse log files
del /F /Q "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\lighthouse_logs\*"
REM - Delete the old plugin bits
del /F /Q "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\plugins\*lighthouse_plugin.py"
rmdir "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\plugins\lighthouse" /s /q
REM - Copy over the new plugin bits
xcopy /s/y "..\plugin\*" "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\plugins\"
del /F /Q "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\plugins\.#lighthouse_plugin.py"
REM - Launch a new IDA session
start "" "C:\tools\disassemblers\IDA 7.4\ida64.exe" "..\..\testcase\ida74\ida64.exe.i64"
REM start "" "C:\tools\disassemblers\IDA 7.4\ida64.exe" "C:\Users\user\Desktop\JavaScriptCore_13.4.i64"
+18
View File
@@ -0,0 +1,18 @@
set LIGHTHOUSE_LOGGING=1
REM - Close any running instances of IDA
call close_IDA.bat
REM - Purge old lighthouse log files
del /F /Q "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\lighthouse_logs\*"
REM - Delete the old plugin bits
del /F /Q "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\plugins\*lighthouse_plugin.py"
rmdir "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\plugins\lighthouse" /s /q
REM - Copy over the new plugin bits
xcopy /s/y "..\plugin\*" "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\plugins\"
del /F /Q "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\plugins\.#lighthouse_plugin.py"
REM - Launch a new IDA session
"C:\tools\disassemblers\IDA 7.0\ida64.exe" "-B" "..\..\testcase\boombox.exe"
-18
View File
@@ -1,18 +0,0 @@
set LIGHTHOUSE_LOGGING=1
REM - Close any running instances of IDA
call close_IDA.bat
REM - Purge old lighthouse log files
del /F /Q "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\lighthouse_logs\*"
REM - Delete the old plugin bits
del /F /Q "C:\tools\disassemblers\IDA 6.8\plugins\*lighthouse_plugin.py"
rmdir "C:\tools\disassemblers\IDA 6.8\plugins\lighthouse" /s /q
REM - Copy over the new plugin bits
xcopy /s/y "..\plugin\*" "C:\tools\disassemblers\IDA 6.8\plugins\"
del /F /Q "C:\tools\disassemblers\IDA 6.8\plugins\.#lighthouse_plugin.py"
REM - Relaunch two IDA sessions
start "" "C:\tools\disassemblers\IDA 6.8\idaq64.exe" "..\..\testcase\boombox.i64"
-18
View File
@@ -1,18 +0,0 @@
set LIGHTHOUSE_LOGGING=1
REM - Close any running instances of IDA
call close_IDA.bat
REM - Purge old lighthouse log files
del /F /Q "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\lighthouse_logs\*"
REM - Delete the old plugin bits
del /F /Q "C:\tools\disassemblers\IDA 6.8\plugins\*lighthouse_plugin.py"
rmdir "C:\tools\disassemblers\IDA 6.8\plugins\lighthouse" /s /q
REM - Copy over the new plugin bits
xcopy /s/y "..\plugin\*" "C:\tools\disassemblers\IDA 6.8\plugins\"
del /F /Q "C:\tools\disassemblers\IDA 6.8\plugins\.#lighthouse_plugin.py"
REM - Relaunch two IDA sessions
start "" "C:\tools\disassemblers\IDA 6.8\idaq.exe" "..\..\testcase\harness_ufs_pdf.instr.idb"
-18
View File
@@ -1,18 +0,0 @@
set LIGHTHOUSE_LOGGING=1
REM - Close any running instances of IDA
call close_IDA.bat
REM - Purge old lighthouse log files
del /F /Q "C:\Users\user\AppData\Roaming\Hex-Rays\IDA Pro\lighthouse_logs\*"
REM - Delete the old plugin bits
del /F /Q "C:\tools\disassemblers\IDA 6.8\plugins\*lighthouse_plugin.py"
rmdir "C:\tools\disassemblers\IDA 6.8\plugins\lighthouse" /s /q
REM - Copy over the new plugin bits
xcopy /s/y "..\plugin\*" "C:\tools\disassemblers\IDA 6.8\plugins\"
del /F /Q "C:\tools\disassemblers\IDA 6.8\plugins\.#lighthouse_plugin.py"
REM - Relaunch two IDA sessions
start "" "C:\tools\disassemblers\IDA 6.8\idaq.exe" "..\..\testcase\idaq.idb"
-65
View File
@@ -1,65 +0,0 @@
import logging
from binaryninja import PluginCommand
from lighthouse.core import Lighthouse
from lighthouse.util.disassembler import disassembler
logger = logging.getLogger("Lighthouse.Binja.Integration")
#------------------------------------------------------------------------------
# Lighthouse Binja Integration
#------------------------------------------------------------------------------
class LighthouseBinja(Lighthouse):
"""
Lighthouse UI Integration for Binary Ninja.
"""
def __init__(self):
super(LighthouseBinja, self).__init__()
def interactive_load_file(self, bv):
disassembler.bv = bv
super(LighthouseBinja, self).interactive_load_file()
def interactive_load_batch(self, bv):
disassembler.bv = bv
super(LighthouseBinja, self).interactive_load_batch()
def interactive_load_batch(self, bv):
disassembler.bv = bv
super(LighthouseBinja, self).open_coverage_overview()
def _install_load_file(self):
PluginCommand.register(
"Lighthouse - Load code coverage file...",
"Load individual code coverage file(s)",
self.interactive_load_file
)
logger.info("Installed the 'Code coverage file' menu entry")
def _install_load_batch(self):
PluginCommand.register(
"Lighthouse - Load code coverage batch...",
"Load and aggregate code coverage files",
self.interactive_load_batch
)
logger.info("Installed the 'Code coverage batch' menu entry")
def _install_open_coverage_overview(self):
PluginCommand.register(
"Lighthouse - Coverage Overview",
"Open the database code coverage overview",
self.interactive_load_batch
)
logger.info("Installed the 'Coverage Overview' menu entry")
# TODO/V35: No good signals to unload (core) plugin on
def _uninstall_load_file(self):
pass
def _uninstall_load_batch(self):
pass
def _uninstall_open_coverage_overview(self):
pass
-497
View File
@@ -1,497 +0,0 @@
import os
import abc
import logging
from lighthouse.ui import CoverageOverview
from lighthouse.util import lmsg
from lighthouse.util.qt import *
from lighthouse.util.disassembler import disassembler
from lighthouse.parsers import DrcovData
from lighthouse.palette import LighthousePalette
from lighthouse.painting import CoveragePainter
from lighthouse.director import CoverageDirector
from lighthouse.coverage import DatabaseCoverage
from lighthouse.metadata import DatabaseMetadata, metadata_progress
logger = logging.getLogger("Lighthouse.Core")
#------------------------------------------------------------------------------
# Plugin Metadata
#------------------------------------------------------------------------------
PLUGIN_VERSION = "0.8.0"
AUTHORS = "Markus Gaasedelen"
DATE = "2018"
#------------------------------------------------------------------------------
# Lighthouse Plugin Core
#------------------------------------------------------------------------------
class Lighthouse(object):
__metaclass__ = abc.ABCMeta
#--------------------------------------------------------------------------
# Initialization
#--------------------------------------------------------------------------
def load(self):
"""
Load the plugin, and integrate its UI into the disassembler.
"""
self._init()
self._install_ui()
# plugin loaded successfully, print the plugin banner
self.print_banner()
logger.info("Successfully loaded plugin")
def _init(self):
"""
Initialize the core components of the plugin.
"""
# the plugin's color palette
self.palette = LighthousePalette()
# the coverage engine
self.director = CoverageDirector(self.palette)
# the coverage painter
self.painter = CoveragePainter(self.director, self.palette)
# the coverage overview widget
self._ui_coverage_overview = None
# the directory to start the coverage file dialog in
self._last_directory = None
def print_banner(self):
"""
Print the plugin banner.
"""
# build the main banner title
banner_params = (PLUGIN_VERSION, AUTHORS, DATE)
banner_title = "Lighthouse v%s - (c) %s - %s" % banner_params
# print plugin banner
lmsg("")
lmsg("-"*75)
lmsg("---[ %s" % banner_title)
lmsg("-"*75)
lmsg("")
#--------------------------------------------------------------------------
# Termination
#--------------------------------------------------------------------------
def unload(self):
"""
Unload the plugin, and remove any UI integrations.
"""
self._uninstall_ui()
self._cleanup()
logger.info("-"*75)
logger.info("Plugin terminated")
def _cleanup(self):
"""
Spin down any lingering core components before plugin unload.
"""
self.painter.terminate()
self.director.terminate()
#--------------------------------------------------------------------------
# UI Integration (Internal)
#--------------------------------------------------------------------------
def _install_ui(self):
"""
Initialize & integrate all plugin UI elements.
"""
self._install_load_file()
self._install_load_batch()
self._install_open_coverage_overview()
def _uninstall_ui(self):
"""
Cleanup & remove all plugin UI integrations.
"""
self._uninstall_open_coverage_overview()
self._uninstall_load_batch()
self._uninstall_load_file()
@abc.abstractmethod
def _install_load_file(self):
"""
Install the 'File->Load->Code coverage file...' menu entry.
"""
pass
@abc.abstractmethod
def _install_load_batch(self):
"""
Install the 'File->Load->Code coverage batch...' menu entry.
"""
pass
@abc.abstractmethod
def _install_open_coverage_overview(self):
"""
Install the 'View->Open subviews->Coverage Overview' menu entry.
"""
pass
@abc.abstractmethod
def _uninstall_load_file(self):
"""
Remove the 'File->Load file->Code coverage file...' menu entry.
"""
pass
@abc.abstractmethod
def _uninstall_load_batch(self):
"""
Remove the 'File->Load file->Code coverage batch...' menu entry.
"""
pass
@abc.abstractmethod
def _uninstall_open_coverage_overview(self):
"""
Remove the 'View->Open subviews->Coverage Overview' menu entry.
"""
pass
#--------------------------------------------------------------------------
# UI Actions (Public)
#--------------------------------------------------------------------------
def open_coverage_overview(self):
"""
Open the dockable 'Coverage Overview' dialog.
"""
self.palette.refresh_colors()
# the coverage overview is already open & visible, simply refresh it
if self._ui_coverage_overview and self._ui_coverage_overview.isVisible():
self._ui_coverage_overview.refresh()
return
# create a new coverage overview if there is not one visible
self._ui_coverage_overview = CoverageOverview(self)
self._ui_coverage_overview.show()
def interactive_load_batch(self):
"""
Perform the user-interactive loading of a coverage batch.
"""
self.palette.refresh_colors()
#
# kick off an asynchronous metadata refresh. this will run in the
# background while the user is selecting which coverage files to load
#
future = self.director.refresh_metadata(
progress_callback=metadata_progress
)
#
# we will now prompt the user with an interactive file dialog so they
# can select the coverage files they would like to load from disk
#
filenames = self._select_coverage_files()
#
# load the selected coverage files from disk (if any), returning a list
# of loaded DrcovData objects (which contain coverage data)
#
drcov_list = load_coverage_files(filenames)
if not drcov_list:
self.director.metadata.abort_refresh()
return
# prompt the user to name the new coverage aggregate
default_name = "BATCH_%s" % self.director.peek_shorthand()
ok, coverage_name = prompt_string(
"Batch Name:",
"Please enter a name for this coverage",
default_name
)
#
# if user didn't enter a name for the batch (or hit cancel) we should
# abort the loading process...
#
if not (ok and coverage_name):
lmsg("User failed to enter a name for the loaded batch...")
self.director.metadata.abort_refresh()
return
#
# to begin mapping the loaded coverage data, we require that the
# asynchronous database metadata refresh has completed. if it is
# not done yet, we will block here until it completes.
#
# a progress dialog depicts the work remaining in the refresh
#
disassembler.show_wait_box("Building database metadata...")
await_future(future)
#
# now that the database metadata is available, we can use the director
# to normalize and condense (aggregate) all the coverage data
#
new_coverage, errors = self.director.aggregate_drcov_batch(drcov_list)
#
# finally, we can inject the aggregated coverage data into the
# director under the user specified batch name
#
disassembler.replace_wait_box("Mapping coverage...")
self.director.create_coverage(coverage_name, new_coverage.data)
# select the newly created batch coverage
disassembler.replace_wait_box("Selecting coverage...")
self.director.select_coverage(coverage_name)
# all done! pop the coverage overview to show the user their results
disassembler.hide_wait_box()
lmsg("Successfully loaded batch %s..." % coverage_name)
self.open_coverage_overview()
# finally, emit any notable issues that occurred during load
warn_errors(errors)
def interactive_load_file(self):
"""
Perform the user-interactive loading of individual coverage files.
"""
self.palette.refresh_colors()
#
# kick off an asynchronous metadata refresh. this will run in the
# background while the user is selecting which coverage files to load
#
future = self.director.refresh_metadata(
progress_callback=metadata_progress
)
#
# we will now prompt the user with an interactive file dialog so they
# can select the coverage files they would like to load from disk
#
filenames = self._select_coverage_files()
#
# load the selected coverage files from disk (if any), returning a list
# of loaded DrcovData objects (which contain coverage data)
#
disassembler.show_wait_box("Loading coverage from disk...")
drcov_list = load_coverage_files(filenames)
if not drcov_list:
disassembler.hide_wait_box()
self.director.metadata.abort_refresh()
return
#
# to begin mapping the loaded coverage data, we require that the
# asynchronous database metadata refresh has completed. if it is
# not done yet, we will block here until it completes.
#
# a progress dialog depicts the work remaining in the refresh
#
disassembler.replace_wait_box("Building database metadata...")
await_future(future)
# insert the loaded drcov data objects into the director
created_coverage, errors = self.director.create_coverage_from_drcov_list(drcov_list)
#
# if the director failed to map any coverage, the user probably
# provided bad files. emit any warnings and bail...
#
if not created_coverage:
lmsg("No coverage files could be loaded...")
disassembler.hide_wait_box()
warn_errors(errors)
return
#
# activate the first of the newly loaded coverage file(s). this is the
# one that will be visible in the coverage overview once opened
#
disassembler.replace_wait_box("Selecting coverage...")
self.director.select_coverage(created_coverage[0])
# all done! pop the coverage overview to show the user their results
disassembler.hide_wait_box()
lmsg("Successfully loaded %u coverage file(s)..." % len(created_coverage))
self.open_coverage_overview()
# finally, emit any notable issues that occurred during load
warn_errors(errors)
#--------------------------------------------------------------------------
# Internal
#--------------------------------------------------------------------------
def _select_coverage_files(self):
"""
Prompt a file selection dialog, returning file selections.
NOTE: This saves & reuses the last known directory for subsequent uses.
"""
if not self._last_directory:
self._last_directory = disassembler.get_database_directory()
# create & configure a Qt File Dialog for immediate use
file_dialog = QtWidgets.QFileDialog(
None,
'Open code coverage file',
self._last_directory,
'All Files (*.*)'
)
file_dialog.setFileMode(QtWidgets.QFileDialog.ExistingFiles)
# prompt the user with the file dialog, and await filename(s)
filenames, _ = file_dialog.getOpenFileNames()
#
# remember the last directory we were in (parsed from a selected file)
# for the next time the user comes to load coverage files
#
if filenames:
self._last_directory = os.path.dirname(filenames[0]) + os.sep
# log the captured (selected) filenames from the dialog
logger.debug("Captured filenames from file dialog:")
for name in filenames:
logger.debug(" - %s" % name)
# return the captured filenames
return filenames
#------------------------------------------------------------------------------
# Util
#------------------------------------------------------------------------------
def load_coverage_files(filenames):
"""
Load multiple code coverage files from disk.
"""
loaded_coverage = []
#
# loop through each of the given filenames and attempt to load/parse
# their coverage data from disk
#
load_failure = False
for filename in filenames:
# attempt to load/parse a single coverage data file from disk
try:
drcov_data = DrcovData(filename)
# catch all for parse errors / bad input / malformed files
except Exception as e:
lmsg("Failed to load coverage %s" % filename)
lmsg(" - Error: %s" % str(e))
logger.exception(" - Traceback:")
load_failure = True
continue
# save the loaded coverage data to the output list
loaded_coverage.append(drcov_data)
# warn if we encountered malformed files...
if load_failure:
warn_drcov_malformed()
# return all the successfully loaded coverage files
return loaded_coverage
#------------------------------------------------------------------------------
# Warnings
#------------------------------------------------------------------------------
def warn_errors(errors):
"""
Warn the user of any encountered errors with a messagebox.
"""
seen = []
for error in errors:
error_type = error[0]
# only emit an error once
if error_type in seen:
continue
# emit relevant error messages
if error_type == CoverageDirector.ERROR_COVERAGE_ABSENT:
warn_module_missing()
elif error_type == CoverageDirector.ERROR_COVERAGE_SUSPICIOUS:
warn_bad_mapping()
else:
raise NotImplementedError("UNKNOWN ERROR OCCURRED")
seen.append(error_type)
def warn_drcov_malformed():
"""
Display a warning for malformed/unreadable coverage files.
"""
disassembler.warning(
"Failed to parse one or more of the selected coverage files!\n\n"
" Possible reasons:\n"
" - You selected a file that was *not* a coverage file.\n"
" - The selected coverage file is malformed or unreadable.\n\n"
"Please see the disassembler console for more info..."
)
def warn_module_missing():
"""
Display a warning for missing coverage data.
"""
disassembler.warning(
"No coverage data was extracted from one of the selected files.\n\n"
" Possible reasons:\n"
" - You selected a coverage file for the wrong binary.\n"
" - The name of the executable file used to generate this database\n"
" is different than the one you collected coverage against.\n\n"
"Please see the disassembler console for more info..."
)
def warn_bad_mapping():
"""
Display a warning for badly mapped coverage data.
"""
disassembler.warning(
"One or more of the loaded coverage files appears to be badly mapped.\n\n"
" Possible reasons:\n"
" - You selected a coverage file that was collected against a\n"
" slightly different version of the binary.\n"
" - You recorded an application with very abnormal control flow.\n"
" - The coverage file might be malformed.\n\n"
"This means that any coverage displayed by Lighthouse is probably\n"
"wrong, and should be used at your own discretion."
)
@@ -1,79 +0,0 @@
import logging
import binaryninja
from binaryninja import HighlightStandardColor
from binaryninja.highlight import HighlightColor
from lighthouse.palette import to_rgb
from lighthouse.painting import DatabasePainter
from lighthouse.util.disassembler import disassembler
logger = logging.getLogger("Lighthouse.Painting.Binja")
#------------------------------------------------------------------------------
# Binary Ninja Painter
#------------------------------------------------------------------------------
class BinjaPainter(DatabasePainter):
"""
Asynchronous Binary Ninja database painter.
"""
PAINTER_SLEEP = 0.01
def __init__(self, director, palette):
super(BinjaPainter, self).__init__(director, palette)
#--------------------------------------------------------------------------
# Paint Primitives
#--------------------------------------------------------------------------
#
# NOTE:
# due to the manner in which Binary Ninja implements basic block
# (node) highlighting, I am not sure it is worth it to paint individual
# instructions. for now we, will simply make the instruction
# painting functions no-op's
#
def _paint_instructions(self, instructions):
self._action_complete.set()
def _clear_instructions(self, instructions):
self._action_complete.set()
def _paint_nodes(self, nodes_coverage):
bv = disassembler.bv
b, g, r = to_rgb(self.palette.coverage_paint)
color = HighlightColor(red=r, green=g, blue=b)
for node_coverage in nodes_coverage:
node_metadata = node_coverage.database._metadata.nodes[node_coverage.address]
for node in bv.get_basic_blocks_starting_at(node_metadata.address):
node.highlight = color
self._painted_nodes.add(node_metadata.address)
self._action_complete.set()
def _clear_nodes(self, nodes_metadata):
bv = disassembler.bv
for node_metadata in nodes_metadata:
for node in bv.get_basic_blocks_starting_at(node_metadata.address):
node.highlight = HighlightStandardColor.NoHighlightColor
self._painted_nodes.discard(node_metadata.address)
self._action_complete.set()
def _refresh_ui(self):
pass
def _cancel_action(self, job):
pass
#--------------------------------------------------------------------------
# Priority Painting
#--------------------------------------------------------------------------
def _priority_paint(self):
current_address = disassembler.get_current_address()
current_function = disassembler.bv.get_function_at(current_address)
if current_function:
self._paint_function(current_function.start)
return True
-474
View File
@@ -1,474 +0,0 @@
import abc
import time
import Queue
import logging
import threading
from lighthouse.util import *
logger = logging.getLogger("Lighthouse.Painting")
class DatabasePainter(object):
"""
An asynchronous disassembler database painting engine.
"""
__metaclass__ = abc.ABCMeta
PAINTER_SLEEP = 0.001
MSG_TERMINATE = 0
MSG_REPAINT = 1
MSG_CLEAR = 2
def __init__(self, director, palette):
#----------------------------------------------------------------------
# Misc
#----------------------------------------------------------------------
self.palette = palette
self._director = director
self._enabled = True
#----------------------------------------------------------------------
# Painted State
#----------------------------------------------------------------------
#
# the coverage painter maintains its own internal record of what
# instruction addresses and graph nodes it has painted.
#
self._painted_nodes = set()
self._painted_instructions = set()
#----------------------------------------------------------------------
# Async
#----------------------------------------------------------------------
#
# to communicate with the asynchronous painting thread, we send a
# a message via the thread event to signal a new paint request, and
# use the repaint_requested bool to interrupt a running paint request.
#
self._action_complete = threading.Event()
self._msg_queue = Queue.Queue()
self._end_threads = False
#
# asynchronous database painting thread
#
self._painting_worker = threading.Thread(
target=self._async_database_painter,
name="DatabasePainter"
)
self._painting_worker.start()
#----------------------------------------------------------------------
# Callbacks
#----------------------------------------------------------------------
# painter callbacks
self._status_changed_callbacks = []
# register for cues from the director
self._director.coverage_switched(self.repaint)
self._director.coverage_modified(self.repaint)
#--------------------------------------------------------------------------
# Status
#--------------------------------------------------------------------------
@property
def enabled(self):
"""
Return the active painting status of the painter.
"""
return self._enabled
def set_enabled(self, status):
"""
Enable or disable the painter.
"""
# enabled/disabled status is not changing, ignore...
if status == self._enabled:
return
lmsg("%s painting..." % ("Enabling" if status else "Disabling"))
self._enabled = status
self.repaint()
# notify listeners that the painter has been enabled/disabled
self._notify_status_changed(status)
#--------------------------------------------------------------------------
# Commands
#--------------------------------------------------------------------------
def terminate(self):
"""
Cleanup & terminate the painter.
"""
self._end_threads = True
self._msg_queue.put(self.MSG_TERMINATE)
self._painting_worker.join()
def repaint(self):
"""
Paint coverage defined by the current database mappings.
"""
if not self.enabled:
return
self._msg_queue.put(self.MSG_REPAINT)
def clear_paint(self):
"""
Clear all paint from the current database (based on metadata)
"""
if self.enabled:
self.set_enabled(False)
self._msg_queue.put(self.MSG_CLEAR)
#--------------------------------------------------------------------------
# Commands
#--------------------------------------------------------------------------
def status_changed(self, callback):
"""
Subscribe a callback for coverage switch events.
"""
register_callback(self._status_changed_callbacks, callback)
def _notify_status_changed(self, status):
"""
Notify listeners of a coverage switch event.
"""
notify_callback(self._status_changed_callbacks, status)
#--------------------------------------------------------------------------
# Paint Primitives
#--------------------------------------------------------------------------
@abc.abstractmethod
def _paint_instructions(self, instructions):
"""
Paint instruction coverage defined by the current database mapping.
"""
pass
@abc.abstractmethod
def _clear_instructions(self, instructions):
"""
Clear paint from the given instructions.
"""
pass
@abc.abstractmethod
def _paint_nodes(self, nodes_coverage):
"""
Paint node coverage defined by the current database mappings.
"""
pass
@abc.abstractmethod
def _clear_nodes(self, nodes_metadata):
"""
Clear paint from the given graph nodes.
"""
pass
@abc.abstractmethod
def _refresh_ui(self):
"""
Refresh the disassembler UI to ensure paint is rendered.
"""
pass
@abc.abstractmethod
def _cancel_action(self, job):
"""
Cancel a paint action using something representing its job.
"""
pass
#------------------------------------------------------------------------------
# Painting - High Level
#------------------------------------------------------------------------------
def _paint_function(self, address):
"""
Paint function instructions & nodes with the current database mappings.
"""
function_metadata = self._director.metadata.functions[address]
function_coverage = self._director.coverage.functions.get(address, None)
if not function_coverage:
return False
#
# ~ compute paint job ~
#
# compute the painted instructions within this function
painted = self._painted_instructions & function_metadata.instructions
# compute the painted instructions that will not get painted over
stale_instructions = painted - function_coverage.instructions
# compute the painted nodes within this function
painted = self._painted_nodes & function_metadata.nodes.viewkeys()
# compute the painted nodes that will not get painted over
stale_nodes_ea = painted - function_coverage.nodes.viewkeys()
stale_nodes = [function_metadata.nodes[ea] for ea in stale_nodes_ea]
# active instructions
instructions = function_coverage.instructions
nodes = function_coverage.nodes.itervalues()
#
# ~ painting ~
#
# clear instructions
if not self._async_action(self._clear_instructions, stale_instructions):
return False # a repaint was requested
# clear nodes
if not self._async_action(self._clear_nodes, stale_nodes):
return False # a repaint was requested
# paint instructions
if not self._async_action(self._paint_instructions, instructions):
return False # a repaint was requested
# paint nodes
if not self._async_action(self._paint_nodes, nodes):
return False # a repaint was requested
# paint finished successfully
return True
def _clear_function(self, address):
"""
Clear paint from the given function.
"""
function_metadata = self._director.metadata.functions[address]
instructions = function_metadata.instructions
nodes = function_metadata.nodes.itervalues()
# clear instructions
if not self._async_action(self._clear_instructions, instructions):
return False # a repaint was requested
# clear nodes
if not self._async_action(self._clear_nodes, nodes):
return False # a repaint was requested
# paint finished successfully
return True
def _paint_database(self):
"""
Repaint the current database based on the current state.
"""
# more code-friendly, readable aliases (db_XX == database_XX)
db_coverage = self._director.coverage
db_metadata = self._director.metadata
start = time.time()
#------------------------------------------------------------------
# immediately paint user-visible regions of the database
if not self._priority_paint():
return False # a repaint was requested
# compute the painted instructions that will not get painted over
stale_inst = self._painted_instructions - db_coverage.coverage
# compute the painted nodes that will not get painted over
stale_nodes_ea = self._painted_nodes - db_coverage.nodes.viewkeys()
stale_nodes = [db_metadata.nodes[ea] for ea in stale_nodes_ea]
# clear old instruction paint
if not self._async_action(self._clear_instructions, stale_inst):
return False # a repaint was requested
# clear old node paint
if not self._async_action(self._clear_nodes, stale_nodes):
return False # a repaint was requested
# paint new instructions
if not self._async_action(self._paint_instructions, db_coverage.coverage):
return False # a repaint was requested
# paint new nodes
if not self._async_action(self._paint_nodes, db_coverage.nodes.itervalues()):
return False # a repaint was requested
#------------------------------------------------------------------
end = time.time()
logger.debug("Full Paint took %s seconds" % (end - start))
logger.debug(" stale_inst: %s" % "{:,}".format(len(stale_inst)))
logger.debug(" fresh inst: %s" % "{:,}".format(len(db_coverage.coverage)))
logger.debug(" stale_nodes: %s" % "{:,}".format(len(stale_nodes)))
logger.debug(" fresh_nodes: %s" % "{:,}".format(len(db_coverage.nodes)))
# paint finished successfully
return True
def _clear_database(self):
"""
Clear all paint from the current database.
"""
db_metadata = self._director.metadata
instructions = db_metadata.instructions
nodes = db_metadata.nodes.viewvalues()
# clear all instructions
if not self._async_action(self._clear_instructions, instructions):
return False # a repaint was requested
# clear all nodes
if not self._async_action(self._clear_nodes, nodes):
return False # a repaint was requested
# paint finished successfully
return True
#--------------------------------------------------------------------------
# Priority Painting
#--------------------------------------------------------------------------
def _priority_paint(self):
"""
Immediately repaint regions of the database visible to the user.
Return True upon completion, or False if interrupted.
"""
return True # NOTE: optional, but recommended
def _priority_paint_functions(self, target_address):
"""
Paint functions in the immediate vicinity of the given address.
This will paint both the instructions & graph nodes of defined functions.
"""
pass # NOTE: optional, organizational
def _priority_paint_instructions(self, target_address, ignore=set()):
"""
Paint instructions in the immediate vicinity of the given address.
Optionally, one can provide a set of addresses to ignore while painting.
"""
pass # NOTE: optional, organizational
#--------------------------------------------------------------------------
# Asynchronous Painting
#--------------------------------------------------------------------------
def _async_database_painter(self):
"""
Asynchronous database painting worker loop.
"""
logger.debug("Starting DatabasePainter thread...")
#
# Asynchronous Database Painting Loop
#
while True:
# wait for the next command to come through
action = self._msg_queue.get()
# repaint the database based on the current state
if action == self.MSG_REPAINT:
result = self._paint_database()
# clear all possible database paint
elif action == self.MSG_CLEAR:
result = self._clear_database()
# spin down the painting thread (this thread)
elif action == self.MSG_TERMINATE:
break
# unknown command
else:
logger.error("UNKNOWN COMMAND! %s" % str(action))
break
# refresh the UI to ensure paint changes are rendered
self._refresh_ui()
# thread exit
logger.debug("Exiting DatabasePainter thread...")
def _async_action(self, paint_action, work_iterable):
"""
Split a normal paint routine into interruptable chunks.
Internal routine for asynchrnous painting.
"""
CHUNK_SIZE = 800 # somewhat arbitrary
# split the given nodes into multiple paints
for work_chunk in chunks(list(work_iterable), CHUNK_SIZE):
#
# reset the paint event signal so that it is ready for the next
# paint request. it will let us know when the asynchrnous paint
# action has completed in the IDA main thread
#
self._action_complete.clear()
#
# paint or unpaint a chunk of 'work' (nodes, or instructions) with
# the given paint function (eg, paint_nodes, clear_instructions)
#
paint_job = paint_action(work_chunk)
#
# wait for the asynchronous paint event to complete or a signal that
# we should end this thread (via end_threads)
#
while not (self._action_complete.wait(timeout=0.1) or self._end_threads):
continue
#
# our end_threads signal/bool can only originate from the main IDA
# thread (plugin termination). we make the assumption that no more
# MFF_WRITE requests (eg, 'paint_action') will get processed.
#
# we do a best effort to cancel the in-flight job (just in case)
# and return so we can exit the thread.
#
if self._end_threads:
self._cancel_action(paint_job)
return False
#
# the operation has been interrupted by a repaint request, bail
# immediately so that we can process the next repaint
#
if not self._msg_queue.empty():
return False
#
# sleep some so we don't choke the main IDA thread
#
time.sleep(self.PAINTER_SLEEP)
# operation completed successfully
return True
-361
View File
@@ -1,361 +0,0 @@
from lighthouse.util.qt import *
from lighthouse.util.disassembler import disassembler
#
# TODO/FUTURE: this file is a huge mess, and will probably be refactored
# whenever I add external theme customization/controls (v0.9?)
#
#------------------------------------------------------------------------------
# Plugin Color Palette
#------------------------------------------------------------------------------
class LighthousePalette(object):
"""
Color Palette for the Lighthouse plugin.
"""
def __init__(self):
"""
Initialize default palette colors for Lighthouse.
"""
# one-time initialization flag, used for selecting initial palette
self._initialized = False
# the active theme name
self._qt_theme = "Dark"
self._disassembly_theme = "Dark"
# the list of available themes
self._themes = \
{
"Dark": 0,
"Light": 1,
}
#
# Coverage Overview
#
self._selection = [QtGui.QColor(100, 0, 130), QtGui.QColor(226, 143, 0)]
self._coverage_none = [QtGui.QColor(30, 30, 30), QtGui.QColor(30, 30, 30)]
self._coverage_bad = [QtGui.QColor(221, 0, 0), QtGui.QColor(207, 31, 0)]
self._coverage_okay = [QtGui.QColor("#bf7ae7"), QtGui.QColor(207, 31, 0)]
self._coverage_good = [QtGui.QColor(51, 153, 255), QtGui.QColor(75, 209, 42)]
#
# IDA Views / HexRays
#
self._coverage_paint = [0x990000, 0xFFE2A8] # NOTE: IDA uses BBGGRR
#
# Composing Shell
#
self._overview_bg = [QtGui.QColor(20, 20, 20), QtGui.QColor(20, 20, 20)]
self._composer_fg = [QtGui.QColor(255, 255, 255), QtGui.QColor(255, 255, 255)]
self._valid_text = [0x80F0FF, 0x0000FF]
self._invalid_text = [0xF02070, 0xFF0000]
self._invalid_highlight = [0x990000, 0xFF0000]
self._shell_hint_bg = [QtGui.QColor(45, 45, 45), QtGui.QColor(45, 45, 45)]
self._shell_hint_fg = [QtGui.QColor(255, 255, 255), QtGui.QColor(255, 255, 255)]
self._combobox_bg = [QtGui.QColor(45, 45, 45), QtGui.QColor(45, 45, 45)]
self._combobox_fg = [QtGui.QColor(255, 255, 255), QtGui.QColor(255, 255, 255)]
self._combobox_selection_bg = [QtGui.QColor(51, 153, 255), QtGui.QColor(51, 153, 255)]
self._combobox_selection_fg = [QtGui.QColor(255, 255, 255), QtGui.QColor(255, 255, 255)]
#
# Composition Grammar
#
self._logic_token = [QtGui.QColor("#F02070"), QtGui.QColor("#FF0000")]
self._comma_token = [QtGui.QColor("#00FF00"), QtGui.QColor("#0000FF")]
self._paren_token = [QtGui.QColor("#40FF40"), QtGui.QColor("#0000FF")]
self._coverage_token = [QtGui.QColor("#80F0FF"), QtGui.QColor("#000000")]
#--------------------------------------------------------------------------
# Theme Management
#--------------------------------------------------------------------------
@property
def disassembly_theme(self):
"""
Return the active IDA theme number.
"""
return self._themes[self._disassembly_theme]
@property
def qt_theme(self):
"""
Return the active Qt theme number.
"""
return self._themes[self._qt_theme]
def refresh_colors(self):
"""
Dynamically compute palette color based on IDA theme.
Depending on if IDA is using a dark or light theme, we *try*
to select colors that will hopefully keep things most readable.
"""
# TODO/FUTURE: temporary until I have a cleaner way to do one-time init
if self._initialized:
return
#
# TODO/THEME:
#
# the dark table (Qt) theme is way better than the light theme
# right now, so we're just going to force that on for everyone
# for the time being.
#
self._qt_theme = "Dark" # self._qt_theme_hint()
self._disassembly_theme = self._disassembly_theme_hint()
# mark the palette as initialized
self._initialized = True
def _disassembly_theme_hint(self):
"""
Binary hint of the IDA color theme.
This routine returns a best effort hint as to what kind of theme is
in use for the IDA Views (Disas, Hex, HexRays, etc).
Returns 'Dark' or 'Light' indicating the user's theme
"""
#
# determine whether to use a 'dark' or 'light' paint based on the
# background color of the user's IDA text based windows
#
bg_color = disassembler.get_disassembly_background_color()
# return 'Dark' or 'Light'
return test_color_brightness(bg_color)
def _qt_theme_hint(self):
"""
Binary hint of the Qt color theme.
This routine returns a best effort hint as to what kind of theme the
QtWdigets throughout IDA are using. This is to accomodate for users
who may be using Zyantific's IDASkins plugins (or others) to further
customize IDA's appearance.
Returns 'Dark' or 'Light' indicating the user's theme
"""
#
# to determine what kind of Qt based theme IDA is using, we create a
# test widget and check the colors put into the palette the widget
# inherits from the application (eg, IDA).
#
test_widget = QtWidgets.QWidget()
#
# in order to 'realize' the palette used to render (draw) the widget,
# it first must be made visible. since we don't want to be popping
# random widgets infront of the user, so we set this attribute such
# that we can silently bake the widget colors.
#
# NOTE/COMPAT: WA_DontShowOnScreen
#
# https://www.riverbankcomputing.com/news/pyqt-56
#
# lmao, don't ask me why they forgot about this attribute from 5.0 - 5.6
#
if USING_PYQT5:
test_widget.setAttribute(103) # taken from http://doc.qt.io/qt-5/qt.html
else:
test_widget.setAttribute(QtCore.Qt.WA_DontShowOnScreen)
# render the (invisible) widget
test_widget.show()
# now we farm the background color from the qwidget
bg_color = test_widget.palette().color(QtGui.QPalette.Window)
# 'hide' & delete the widget
test_widget.hide()
test_widget.deleteLater()
# return 'Dark' or 'Light'
return test_color_brightness(bg_color)
#--------------------------------------------------------------------------
# Coverage Overview
#--------------------------------------------------------------------------
@property
def selection(self):
return self._selection[self.qt_theme]
@property
def coverage_none(self):
return self._coverage_none[self.qt_theme]
@property
def coverage_bad(self):
return self._coverage_bad[self.qt_theme]
@property
def coverage_okay(self):
return self._coverage_okay[self.qt_theme]
@property
def coverage_good(self):
return self._coverage_good[self.qt_theme]
#--------------------------------------------------------------------------
# IDA Views / HexRays
#--------------------------------------------------------------------------
@property
def coverage_paint(self):
return self._coverage_paint[self.disassembly_theme]
#--------------------------------------------------------------------------
# Composing Shell
#--------------------------------------------------------------------------
@property
def overview_bg(self):
return self._overview_bg[self.qt_theme]
@property
def composer_fg(self):
return self._composer_fg[self.qt_theme]
@property
def valid_text(self):
return self._valid_text[self.qt_theme]
@property
def invalid_text(self):
return self._invalid_text[self.qt_theme]
@property
def invalid_highlight(self):
return self._invalid_highlight[self.qt_theme]
@property
def shell_hint_bg(self):
return self._shell_hint_bg[self.qt_theme]
@property
def shell_hint_fg(self):
return self._shell_hint_fg[self.qt_theme]
#--------------------------------------------------------------------------
# Coverage Combobox
#--------------------------------------------------------------------------
@property
def combobox_bg(self):
return self._combobox_bg[self.qt_theme]
@property
def combobox_fg(self):
return self._combobox_fg[self.qt_theme]
@property
def combobox_selection_bg(self):
return self._combobox_selection_bg[self.qt_theme]
@property
def combobox_selection_fg(self):
return self._combobox_selection_fg[self.qt_theme]
#--------------------------------------------------------------------------
# Composition Grammar
#--------------------------------------------------------------------------
@property
def logic_token(self):
return self._logic_token[self.qt_theme]
@property
def comma_token(self):
return self._comma_token[self.qt_theme]
@property
def paren_token(self):
return self._paren_token[self.qt_theme]
@property
def coverage_token(self):
return self._coverage_token[self.qt_theme]
@property
def TOKEN_COLORS(self):
"""
Return the palette of token colors.
"""
return \
{
# logic operators
"OR": self.logic_token,
"XOR": self.logic_token,
"AND": self.logic_token,
"MINUS": self.logic_token,
# misc
"COMMA": self.comma_token,
"LPAREN": self.paren_token,
"RPAREN": self.paren_token,
#"WS": self.whitepsace_token,
#"UNKNOWN": self.unknown_token,
# coverage
"COVERAGE_TOKEN": self.coverage_token,
}
#------------------------------------------------------------------------------
# Palette Util
#------------------------------------------------------------------------------
def to_rgb(color):
return ((color >> 16 & 0xFF), (color >> 8 & 0xFF), (color & 0xFF))
def test_color_brightness(color):
"""
Test the brightness of a color.
"""
if color.lightness() > 255.0/2:
return "Light"
else:
return "Dark"
def compute_color_on_gradiant(percent, color1, color2):
"""
Compute the color specified by a percent between two colors.
TODO/PERF: This is silly, heavy, and can be refactored.
"""
# dump the rgb values from QColor objects
r1, g1, b1, _ = color1.getRgb()
r2, g2, b2, _ = color2.getRgb()
# compute the new color across the gradiant of color1 -> color 2
r = r1 + percent * (r2 - r1)
g = g1 + percent * (g2 - g1)
b = b1 + percent * (b2 - b1)
# return the new color
return QtGui.QColor(r,g,b)
-1
View File
@@ -1 +0,0 @@
from drcov import DrcovData
-1
View File
@@ -1 +0,0 @@
from .coverage_overview import CoverageOverview
-71
View File
@@ -1,71 +0,0 @@
import cProfile
#------------------------------------------------------------------------------
# Debug
#------------------------------------------------------------------------------
#
# This file contains random snippets of code that I frequently use while
# developing and debugging parts of lighthouse. I don't expect any of this
# code to be active or in use for major releases.
#
#------------------------------------------------------------------------------
# Call Profiling
#------------------------------------------------------------------------------
pr = cProfile.Profile()
def profile(func):
"""
A simple function profiling decorator.
"""
def wrap(*args, **kwargs):
global pr
pr.enable()
result = func(*args, **kwargs)
pr.disable()
pr.print_stats(sort="tottime")
return result
return wrap
#------------------------------------------------------------------------------
# Function Line Profiling
#------------------------------------------------------------------------------
# from: https://gist.github.com/sibelius/3920b3eb5adab482b105
try:
from line_profiler import LineProfiler
def line_profile(func):
def profiled_func(*args, **kwargs):
try:
profiler = LineProfiler()
profiler.add_function(func)
profiler.enable_by_count()
return func(*args, **kwargs)
finally:
profiler.print_stats()
return profiled_func
except ImportError:
def line_profile(func):
def nothing(*args, **kwargs):
return func(*args, **kwargs)
return nothing
#------------------------------------------------------------------------------
# Module Line Profiling
#------------------------------------------------------------------------------
if False:
from line_profiler import LineProfiler
lpr = LineProfiler()
# change this to the target file / module to profile
import lighthouse.metadata as metadata
lpr.add_module(metadata)
# put this code somewhere to dump results:
#global lpr
#lpr.enable_by_count()
#lpr.disable_by_count()
#lpr.print_stats(stripzeros=True)
@@ -1,447 +0,0 @@
# -*- coding: utf-8 -*-
import os
import sys
import logging
import functools
import threading
import binaryninja
from binaryninja import PythonScriptingInstance, binaryview
from binaryninja.plugin import BackgroundTaskThread
#------------------------------------------------------------------------------
# External PyQt5 Dependency
#------------------------------------------------------------------------------
#
# amend the Python import path with a Libs folder for additional pip
# packages required by Lighthouse (at least on Windows, and maybe macOS)
#
# TODO/FUTURE: it is kind of dirty that we have to do this here. maybe it
# can be moved with a later refactor. in the long run, binary ninja will
# ship with PyQt5 bindings in-box.
#
DEPENDENCY_PATH = os.path.join(
binaryninja.user_plugin_path,
"Lib",
"site-packages"
)
sys.path.append(DEPENDENCY_PATH)
#------------------------------------------------------------------------------
from .api import DisassemblerAPI, DockableShim
from ..qt import *
from ..misc import is_mainthread, not_mainthread
logger = logging.getLogger("Lighthouse.API.Binja")
#------------------------------------------------------------------------------
# Utils
#------------------------------------------------------------------------------
def execute_sync(function):
"""
Synchronize with the disassembler for safe database access.
"""
@functools.wraps(function)
def wrapper(*args, **kwargs):
#
# in Binary Ninja, it is only safe to access the BNDB from a thread
# that is *not* the mainthread. if we appear to already be in a
# background thread of some sort, simply execute the given function
#
if not is_mainthread():
return function(*args, **kwargs)
#
# if we are in the mainthread, we need to schedule a background
# task to perform our database task/function instead
#
# this inline function definition is technically what will execute
# in a database-safe background thread. we use this thunk to
# capture any output the function may want to return to the user.
#
output = [None]
def thunk():
output[0] = function(*args, **kwargs)
return 1
class DatabaseRead(BackgroundTaskThread):
"""
A stub task to safely read from the BNDB.
"""
def __init__(self, text, function):
super(DatabaseRead, self).__init__(text, False)
self._task_to_run = function
def run(self):
self._task_to_run()
self.finish()
# schedule the databases read and wait for its completion
t = DatabaseRead("Accessing database...", thunk)
t.start()
t.join()
# return the output of the synchronized execution / read
return output[0]
return wrapper
#------------------------------------------------------------------------------
# Disassembler API
#------------------------------------------------------------------------------
class BinjaAPI(DisassemblerAPI):
"""
The Binary Ninja implementation of the disassembler API abstraction.
"""
NAME = "BINJA"
def __init__(self, bv=None):
super(BinjaAPI, self).__init__()
self._init_version()
# binja specific amenities
self._bv = bv
self._python = _binja_get_scripting_instance()
def _init_version(self):
# retrieve Binja's version #
disassembler_version = binaryninja.core_version.split("-", 1)[0]
major, minor, patch = map(int, disassembler_version.split("."))
# save the version number components for later use
self._version_major = major
self._version_minor = minor
self._version_patch = patch
#--------------------------------------------------------------------------
# Properties
#--------------------------------------------------------------------------
@property
def bv(self):
return self._bv
@bv.setter
def bv(self, bv):
if self._bv == bv:
return
if self._bv:
raise ValueError("BinaryView cannot be changed once set...")
self._bv = bv
@property
def version_major(self):
return self._version_major
@property
def version_minor(self):
return self._version_minor
@property
def version_patch(self):
return self._version_patch
@property
def headless(self):
return not binaryninja.core_ui_enabled
#--------------------------------------------------------------------------
# Synchronization Decorators
#--------------------------------------------------------------------------
@staticmethod
def execute_read(function):
return execute_sync(function)
@staticmethod
def execute_write(function):
return execute_sync(function)
@staticmethod
def execute_ui(function):
@functools.wraps(function)
def wrapper(*args, **kwargs):
ff = functools.partial(function, *args, **kwargs)
# if we are already in the main (UI) thread, execute now
if is_mainthread():
ff()
return
# schedule the task to run in the main thread
try:
binaryninja.execute_on_main_thread(ff)
except AttributeError: # TODO/V35: binja bug, fixed on dev
pass
return wrapper
#--------------------------------------------------------------------------
# API Shims
#--------------------------------------------------------------------------
#
# NOTE/TODO/V35:
#
# The use of @not_mainthread or @execute_read on some of these API's
# is to ensure the function is called from a background thread/task.
# This is because calling database functions from the mainthread can
# cause deadlocks (not threadsafe?) in Binary Ninja...
#
# this is pretty annoying because it conflicts directly with IDA
# which *needs* database accesses to be made from the mainthread
#
def create_rename_hooks(self):
return RenameHooks(self.bv)
def get_current_address(self):
if not self._python:
self._python = _binja_get_scripting_instance()
if not self._python:
return -1
return self._python.current_addr
@execute_read.__func__
def get_database_directory(self):
return os.path.dirname(self.bv.file.filename)
def get_disassembler_user_directory(self):
return os.path.split(binaryninja.user_plugin_path)[0]
@not_mainthread
def get_function_addresses(self):
return [x.start for x in self.bv.functions]
@not_mainthread
def get_function_name_at(self, address):
func = self.bv.get_function_at(address)
if not func:
return None
return func.symbol.short_name
@execute_read.__func__
def get_function_raw_name_at(self, address):
func = self.bv.get_function_at(address)
if not func:
return None
return func.name
@not_mainthread
def get_imagebase(self):
return self.bv.start
@not_mainthread
def get_root_filename(self):
return os.path.basename(self.bv.file.original_filename)
def navigate(self, address):
return self.bv.navigate(self.bv.view, address)
@execute_write.__func__
def set_function_name_at(self, function_address, new_name):
func = self.bv.get_function_at(function_address)
if not func:
return
if new_name == "":
new_name = None
func.name = new_name
#
# TODO/V35: As a workaround for no symbol events, we trigger a data
# notification for this function instead.
#
self.bv.write(function_address, self.bv.read(function_address, 1))
#--------------------------------------------------------------------------
# UI API Shims
#--------------------------------------------------------------------------
def get_disassembly_background_color(self):
palette = QtGui.QPalette()
return palette.color(QtGui.QPalette.Button)
def is_msg_inited(self):
return True
def warning(self, text):
binaryninja.interaction.show_message_box("Warning", text)
#------------------------------------------------------------------------------
# Function Prefix API
#------------------------------------------------------------------------------
PREFIX_SEPARATOR = "" # Unicode 0x2581
#------------------------------------------------------------------------------
# Hooking
#------------------------------------------------------------------------------
class RenameHooks(object):
"""
A Hooking class to catch function renames in Binary Ninja.
"""
def __init__(self, bv):
self._bv = bv
# hook certain Binary Ninja notifications
self._hooks = binaryview.BinaryDataNotification()
self._hooks.function_updated = self._workaround
# TODO/V35: turns out there are no adequate symbol event hooks...
#self._hooks.function_update_requested = self._before
#self._hooks.function_updated = self._after
#self._names = {}
def hook(self):
self._bv.register_notification(self._hooks)
def unhook(self):
self._bv.unregister_notification(self._hooks)
@BinjaAPI.execute_ui
def _renamed(self, address, new_name):
"""
Pass off the (internal) rename event to the mainthread.
"""
self.renamed(address, new_name)
def _before(self, _, function):
"""
Capture function name prior to modification.
"""
self._names[function.start] = function.name
def _after(self, _, function):
"""
Capture function name post modification
"""
#
# if we don't have an old name for a given function logged, that
# means we must have missed the function_update_requested event for it.
#
# hopefully this should never happen during real *rename* events...
#
old_name = self._names.get(function.start, None)
if not old_name:
return
# if the function name hasn't changed, then there is nothing to do!
if old_name == function.name:
return
# fire our custom 'function renamed' event
self._renamed(function.start, function.name)
#--------------------------------------------------------------------------
# Temporary Workaound
#--------------------------------------------------------------------------
def _workaround(self, _, function):
"""
TODO/V35: workaround to detect name changes pending better API's
"""
function_metadata = self.metadata.get_function(function.start)
if not function_metadata:
return
# if the function name hasn't changed, then there is nothing to do!
if function_metadata.name == function.symbol.short_name:
return
# fire our custom 'function renamed' event
self._renamed(function.start, function.symbol.short_name)
#------------------------------------------------------------------------------
# UI
#------------------------------------------------------------------------------
class DockableWindow(DockableShim):
"""
A dockable Qt widget for Binary Ninja.
"""
def __init__(self, window_title, icon_path):
super(DockableWindow, self).__init__(window_title, icon_path)
# configure dockable widget container
self._main_window = get_qt_main_window()
self._widget = QtWidgets.QWidget()
self._dockable = QtWidgets.QDockWidget(window_title, self._main_window)
self._dockable.setWidget(self._widget)
self._dockable.setWindowIcon(self._window_icon)
self._dockable.setAttribute(QtCore.Qt.WA_DeleteOnClose)
self._dockable.setSizePolicy(
QtWidgets.QSizePolicy.Expanding,
QtWidgets.QSizePolicy.Expanding
)
self._widget.setSizePolicy(
QtWidgets.QSizePolicy.Expanding,
QtWidgets.QSizePolicy.Expanding
)
# dock the widget on the right side of Binja
self._main_window.addDockWidget(
QtCore.Qt.RightDockWidgetArea,
self._dockable
)
def show(self):
#
# NOTE/HACK:
# this is a little dirty, but is used to set the default width
# of the coverage overview / dockable widget when it is first shown
#
default_width = self._widget.sizeHint().width()
self._dockable.setMinimumWidth(default_width)
# show the widget
self._dockable.show()
# undo the HACK
self._dockable.setMinimumWidth(0)
#------------------------------------------------------------------------------
# Binary Ninja Hacks XXX / TODO / V35
#------------------------------------------------------------------------------
def _binja_get_scripting_instance():
"""
Get the python scripting console in Binary Ninja.
"""
for t in threading.enumerate():
if type(t) == PythonScriptingInstance.InterpreterThread:
return t
return None
def binja_get_bv():
"""
Get the current BinaryView in Binary Ninja.
"""
python = _binja_get_scripting_instance()
if not python:
return None
return python.current_view
def binja_get_function_at(address):
"""
Get the function object at the given address.
"""
bv = binja_get_bv()
if not bv:
return None
return bv.get_function_at(address)
-68
View File
@@ -1,68 +0,0 @@
#
# this global is used to indicate whether Qt bindings for python are present
# and available for use by Lighthouse.
#
QT_AVAILABLE = False
#------------------------------------------------------------------------------
# PyQt5 <--> PySide (Qt4) Interoperability
#------------------------------------------------------------------------------
#
# from Qt4 --> Qt5, a number of objects / modules have changed places
# within the Qt codebase. we use this file to shim/re-alias a few of these
# changes to reduce the number of compatibility checks / code churn in the
# plugin code that consumes them.
#
# this makes the plugin codebase compatible with both PySide & PyQt5, a
# necessary requirement to maintain compatibility with IDA 6.8 --> 7.x
#
# additionally, the 'USING_PYQT5' global can be used to check if we are
# running in a PyQt5 context (versus PySide/Qt4). This may be used in a few
# places throughout the project that could not be covered by our shims.
#
USING_PYQT5 = False
#------------------------------------------------------------------------------
# PyQt5 Compatibility
#------------------------------------------------------------------------------
# attempt to load PyQt5
if QT_AVAILABLE == False:
try:
import PyQt5.QtGui as QtGui
import PyQt5.QtCore as QtCore
import PyQt5.QtWidgets as QtWidgets
# importing went okay, PyQt5 must be available for use
QT_AVAILABLE = True
USING_PYQT5 = True
# import failed, PyQt5 is not available
except ImportError:
pass
#------------------------------------------------------------------------------
# PySide Compatibility
#------------------------------------------------------------------------------
# if PyQt5 did not import, try to load PySide
if QT_AVAILABLE == False:
try:
import PySide.QtGui as QtGui
import PySide.QtCore as QtCore
# alias for less PySide <--> PyQt5 shimming
QtWidgets = QtGui
QtCore.pyqtSignal = QtCore.Signal
QtCore.pyqtSlot = QtCore.Slot
# importing went okay, PySide must be available for use
QT_AVAILABLE = True
# import failed. No Qt / UI bindings available...
except ImportError:
pass
+1
View File
@@ -0,0 +1 @@
get_context = lambda x: None
@@ -34,6 +34,7 @@ class TextToken(object):
# NOTE: this is now dynamically computed in parse(...)
#COVERAGE_TOKEN = r'(?P<COVERAGE_TOKEN>[A-Za-z])'
AGGREGATE_TOKEN = '*'
#
# LOGIC_TOKEN:
@@ -117,21 +118,6 @@ class TokenLogicOperator(AstToken):
return operator.sub
raise ValueError("Unknown Operator")
class TokenCoverageRange(AstToken):
"""
AST Token for a coverage range reference.
eg: 'A,Z'
"""
def __init__(self, start, comma, end):
super(TokenCoverageRange, self).__init__()
self.text_tokens = [start, comma, end]
# referenced coverage sets
self.symbol_start = start.value.upper()
self.symbol_end = end.value.upper()
class TokenCoverageSingle(AstToken):
"""
AST Token for a single coverage reference.
@@ -182,28 +168,25 @@ def _ast_equal_recursive(first, second):
if type(first) != type(second):
return False
#
# if both tokens are terminating / None, they are a match
#
if first == second == None:
return True
#
# if the current node is a logic operator, we need to evaluate the
# expressions that make up its input.
#
if isinstance(first, TokenLogicOperator):
elif isinstance(first, TokenLogicOperator):
if not _ast_equal_recursive(first.op1, second.op1):
return False
if not _ast_equal_recursive(first.op2, second.op2):
return False
return first.operator == second.operator
#
# if the current node is a coverage range, we need to evaluate the
# range expression. this will produce an aggregate coverage set
# described by the start/end of the range (Eg, 'A,D')
#
elif isinstance(first, TokenCoverageRange):
return first.symbol_start == second.symbol_start and \
first.symbol_end == second.symbol_end
#
# if the current node is a coverage token, we need simply need
# to compare its symbol.
@@ -216,7 +199,7 @@ def _ast_equal_recursive(first, second):
# unknown token? (this should never happen)
#
raise False
raise ValueError("Unknown token types, cannot compare them...")
#------------------------------------------------------------------------------
# Parsing
@@ -273,17 +256,11 @@ class CompositionParser(object):
EXPRESSION:
'(' EXPRESSION ')' COMPOSITION_TAIL | COVERAGE COMPOSITION_TAIL
COVERAGE:
COVERAGE_TOKEN COVERAGE_RANGE
COVERAGE_RANGE:
',' COVERAGE_TOKEN | None
COVERAGE_TOKEN:
'A' | 'B' | 'C' | ... | 'Z'
'A' | 'B' | 'C' | ... | 'Z' | AGGREGATE_TOKEN
LOGIC_TOKEN:
'&' | '|' | '^' | '-'
'&' | '|' | '^' | '-' | None
"""
@@ -310,7 +287,7 @@ class CompositionParser(object):
# reflect the state of loaded coverage
#
COVERAGE_TOKEN = r'(?P<COVERAGE_TOKEN>[%s])' % ''.join(coverage_tokens)
COVERAGE_TOKEN = r'(?P<COVERAGE_TOKEN>[%s])' % ''.join(coverage_tokens + [AGGREGATE_TOKEN])
#
# if there were any coverage tokens defined, then we definitely need
@@ -420,7 +397,7 @@ class CompositionParser(object):
def _EXPRESSION(self):
"""
EXPRESSION:
'(' EXPRESSION ')' COMPOSITION_TAIL | COVERAGE COMPOSITION_TAIL
'(' EXPRESSION ')' COMPOSITION_TAIL | COVERAGE_TOKEN COMPOSITION_TAIL
"""
#
@@ -449,49 +426,24 @@ class CompositionParser(object):
#
else:
expression = self._COVERAGE()
expression = self._COVERAGE_TOKEN()
# ... [COMPOSITION_TAIL]
return self._COMPOSITION_TAIL(expression)
def _COVERAGE(self):
"""
COVERAGE:
COVERAGE_TOKEN COVERAGE_RANGE
"""
coverage_start = self._COVERAGE_TOKEN()
coverage_range = self._COVERAGE_RANGE()
# if a there was a trailing ',A-Za-z' parsed, it's a coverage range
if coverage_range:
comma, coverage_end = coverage_range
return TokenCoverageRange(coverage_start, comma, coverage_end)
# return a single coverage set
return TokenCoverageSingle(coverage_start)
def _COVERAGE_RANGE(self):
"""
COVERAGE_RANGE:
',' COVERAGE_TOKEN | None
"""
if self._accept("COMMA"):
return (self.current_token, self._COVERAGE_TOKEN())
return None
def _COVERAGE_TOKEN(self):
"""
COVERAGE_TOKEN:
'A' | 'B' | 'C' | ... | 'Z'
'A' | 'B' | 'C' | ... | 'Z' | AGGREGATE_TOKEN
"""
if self._accept("COVERAGE_TOKEN"):
return self.current_token
return TokenCoverageSingle(self.current_token)
self._parse_error("Expected COVERAGE_TOKEN", TokenCoverageSingle)
def _LOGIC_TOKEN(self):
"""
LOGIC_TOKEN:
'&' | '|' | '^' | '-'
'&' | '|' | '^' | '-' | None
"""
if self._accept("OR") or \
self._accept("XOR") or \
@@ -21,13 +21,13 @@ class ComposingShell(QtWidgets.QWidget):
independent, but obviously must communicate with the director.
"""
def __init__(self, director, table_model, table_view=None):
def __init__(self, lctx, table_model, table_view=None):
super(ComposingShell, self).__init__()
self.setObjectName(self.__class__.__name__)
# external entities
self._director = director
self._palette = director._palette
self._director = lctx.director
self._palette = lctx.palette
self._table_model = table_model
self._table_view = table_view
@@ -46,6 +46,7 @@ class ComposingShell(QtWidgets.QWidget):
# configure the widget for use
self._ui_init()
self.refresh_theme()
#--------------------------------------------------------------------------
# Properties
@@ -69,7 +70,7 @@ class ComposingShell(QtWidgets.QWidget):
# initialize a monospace font to use with our widget(s)
self._font = MonospaceFont()
self._font.setPointSizeF(normalize_to_dpi(9))
self._font.setPointSizeF(normalize_to_dpi(10))
self._font_metrics = QtGui.QFontMetricsF(self._font)
# initialize our ui elements
@@ -86,30 +87,18 @@ class ComposingShell(QtWidgets.QWidget):
# the composer label at the head of the shell
self._line_label = QtWidgets.QLabel("Composer")
self._line_label.setStyleSheet("QLabel { margin: 0 1ex 0 1ex }")
self._line_label.setAlignment(QtCore.Qt.AlignVCenter | QtCore.Qt.AlignHCenter)
self._line_label.setAlignment(QtCore.Qt.AlignCenter)
self._line_label.setFont(self._font)
self._line_label.setFixedWidth(self._line_label.sizeHint().width())
# the text box / shell / ComposingLine
self._line = ComposingLine()
# configure the shell background & default text color
palette = self._line.palette()
palette.setColor(QtGui.QPalette.Base, self._palette.overview_bg)
palette.setColor(QtGui.QPalette.Text, self._palette.composer_fg)
palette.setColor(QtGui.QPalette.WindowText, self._palette.composer_fg)
self._line.setPalette(palette)
def _ui_init_completer(self):
"""
Initialize the coverage hint UI elements.
"""
# NOTE/COMPAT:
if USING_PYQT5:
self._completer_model = QtCore.QStringListModel([])
else:
self._completer_model = QtGui.QStringListModel([])
self._completer_model = QtCore.QStringListModel([])
self._completer = QtWidgets.QCompleter(self)
self._completer.setCompletionMode(QtWidgets.QCompleter.PopupCompletion)
@@ -118,10 +107,6 @@ class ComposingShell(QtWidgets.QWidget):
self._completer.setModel(self._completer_model)
self._completer.setWrapAround(False)
self._completer.popup().setFont(self._font)
self._completer.popup().setStyleSheet(
"background: %s;" % self._palette.shell_hint_bg.name() +
"color: %s;" % self._palette.shell_hint_fg.name()
)
self._completer.setWidget(self._line)
def _ui_init_signals(self):
@@ -142,6 +127,7 @@ class ComposingShell(QtWidgets.QWidget):
self._director.coverage_created(self._internal_refresh)
self._director.coverage_deleted(self._internal_refresh)
self._director.coverage_modified(self._internal_refresh)
self._director.coverage_switched(self._coverage_switched)
# register for cues from the model
self._table_model.layoutChanged.connect(self._ui_shell_text_changed)
@@ -176,12 +162,44 @@ class ComposingShell(QtWidgets.QWidget):
"""
self._internal_refresh()
@disassembler.execute_ui
def refresh_theme(self):
"""
Refresh UI facing elements to reflect the current theme.
"""
assert (self._line and self._completer), "UI not yet initialized..."
# configure the shell background & default text color
qpal = self._line.palette()
qpal.setColor(QtGui.QPalette.Text, self._palette.shell_text)
qpal.setColor(QtGui.QPalette.WindowText, self._palette.shell_text)
self._line.setPalette(qpal)
# set other hard to access shell theme elements
self._line.setStyleSheet(
"QPlainTextEdit {"
" color: %s;" % self._palette.shell_text.name() + # this line ensures the text cursor changes color, with the theme
" background-color: %s;" % self._palette.shell_background.name() +
" border: 1px solid %s;" % self._palette.shell_border.name() +
"} "
"QPlainTextEdit:hover, QPlainTextEdit:focus {"
" border: 1px solid %s;" % self._palette.shell_border_focus.name() +
"}"
)
# refresh completer popup style...
self._completer.popup().setStyleSheet(
"background: %s;" % self._palette.shell_hint_background.name() +
"color: %s;" % self._palette.shell_hint_text.name()
)
@disassembler.execute_ui
def _internal_refresh(self):
"""
Internal refresh of the shell.
"""
self._refresh_hint_list()
self._ui_shell_text_changed()
def _refresh_hint_list(self):
"""
@@ -203,6 +221,22 @@ class ComposingShell(QtWidgets.QWidget):
# queue a UI coverage hint if necessary
self._ui_hint_coverage_refresh()
def _coverage_switched(self):
"""
Handle a coverage switched event.
specifically, we want cover the specical case where the hot shell is
being switched to. In these cases, we should forcefully clear the
'last' AST so that the full shell expression is re-evaluated and
sent forward to the director.
this will ensure that the director will evaluate and display the
results of the present expression as the 'Hot Shell' is now active.
"""
if self._director.coverage_name == "Hot Shell":
self._last_ast = None
self._internal_refresh()
#--------------------------------------------------------------------------
# Signal Handlers
#--------------------------------------------------------------------------
@@ -383,9 +417,9 @@ class ComposingShell(QtWidgets.QWidget):
# color search based on if there are any matching results
if self._table_model.rowCount():
self._color_text(self._palette.valid_text, start=1)
self._color_text(self._palette.shell_text_valid, start=1)
else:
self._color_text(self._palette.invalid_text, start=1)
self._color_text(self._palette.shell_text_invalid, start=1)
################# UPDATES ENABLED #################
self._line.setUpdatesEnabled(True)
@@ -440,9 +474,9 @@ class ComposingShell(QtWidgets.QWidget):
except ValueError:
pass
else:
function_metadata = self._director.metadata.get_function(address)
if function_metadata:
return function_metadata.address
functions = self._director.metadata.get_functions_containing(address)
if functions:
return functions[0].address
#
# the user string did not translate to a parsable hex number (address)
@@ -454,16 +488,29 @@ class ComposingShell(QtWidgets.QWidget):
# special case to make 'sub_*' prefixed user inputs case insensitive
if text.lower().startswith("sub_"):
text = "sub_" + text[4:].upper()
# look up the text function name within the director's metadata
# attempt uppercase hex (IDA...)
function_metadata = self._director.metadata.get_function_by_name("sub_" + text[4:].upper())
if function_metadata:
return function_metadata.address
# attempt lowercase hex (Binja...)
function_metadata = self._director.metadata.get_function_by_name("sub_" + text[4:].lower())
if function_metadata:
return function_metadata.address
#
# no luck yet, let's just throw the user's raw text at the lookup. this
# would probably be a function they renamed, such as 'foobar'
#
function_metadata = self._director.metadata.get_function_by_name(text)
if function_metadata:
return function_metadata.address
#
# the user string did not translate to a function name that could
# be found in the director.
# be found in the director. so I guess they're not trying to jump...
#
# failure, the user input (text) isn't a jump ...
@@ -498,7 +545,7 @@ class ComposingShell(QtWidgets.QWidget):
self._color_clear()
# color jump
self._color_text(self._palette.valid_text)
self._color_text(self._palette.shell_text_valid)
################# UPDATES ENABLED #################
self._line.setUpdatesEnabled(True)
@@ -653,6 +700,18 @@ class ComposingShell(QtWidgets.QWidget):
self._ui_hint_coverage_hide()
return
#
# if the text cursor is moving and the user has their left mouse
# button held, then they are probably doing a click + drag text
# selection so we shouldn't be naggin them with hints and stuff
#
# without this condition, click+drag selection gets really choppy
#
if QtWidgets.QApplication.mouseButtons() & QtCore.Qt.LeftButton:
self._ui_hint_coverage_hide()
return
# scrape info from the current shell text state
cursor_index = self._line.textCursor().position()
text_token = self._get_cursor_coverage_token(cursor_index)
@@ -850,10 +909,10 @@ class ComposingShell(QtWidgets.QWidget):
cursor_position = cursor.position()
# setup the invalid text highlighter
invalid_color = self._palette.invalid_highlight
invalid_color = self._palette.shell_highlight_invalid
highlight = QtGui.QTextCharFormat()
highlight.setFontWeight(QtGui.QFont.Bold)
highlight.setBackground(QtGui.QBrush(QtGui.QColor(invalid_color)))
highlight.setBackground(QtGui.QBrush(invalid_color))
self._line.blockSignals(True)
################# UPDATES DISABLED #################
@@ -902,7 +961,7 @@ class ComposingShell(QtWidgets.QWidget):
# setup a simple font coloring (or clearing) text format
simple = QtGui.QTextCharFormat()
if color:
simple.setForeground(QtGui.QBrush(QtGui.QColor(color)))
simple.setForeground(QtGui.QBrush(color))
self._line.blockSignals(True)
################# UPDATES DISABLED #################
@@ -970,7 +1029,7 @@ class ComposingLine(QtWidgets.QPlainTextEdit):
# initialize a monospace font to use with our widget(s)
self._font = MonospaceFont()
self._font.setPointSizeF(normalize_to_dpi(9))
self._font.setPointSizeF(normalize_to_dpi(10))
self._font_metrics = QtGui.QFontMetricsF(self._font)
self.setFont(self._font)
@@ -986,7 +1045,7 @@ class ComposingLine(QtWidgets.QPlainTextEdit):
# set the height of the textbox based on some arbitrary math :D
LINE_PADDING = self.document().documentMargin()*2
line_height = self._font_metrics.height() + LINE_PADDING + 2
self.setFixedHeight(line_height)
self.setFixedHeight(int(line_height))
#--------------------------------------------------------------------------
# QPlainTextEdit Overloads
+111
View File
@@ -0,0 +1,111 @@
import os
import logging
from lighthouse.util.qt import *
from lighthouse.painting import CoveragePainter
from lighthouse.director import CoverageDirector
from lighthouse.coverage import DatabaseCoverage
from lighthouse.metadata import DatabaseMetadata
from lighthouse.util.disassembler import disassembler, DisassemblerContextAPI
logger = logging.getLogger("Lighthouse.Context")
#------------------------------------------------------------------------------
# Lighthouse Session Context
#------------------------------------------------------------------------------
class LighthouseContext(object):
"""
A database/binary-unique instance of Lighthouse and its subsystems.
"""
def __init__(self, core, dctx):
disassembler[self] = DisassemblerContextAPI(dctx)
self.core = core
self.dctx = dctx
self._started = False
# the database metadata cache
self.metadata = DatabaseMetadata(self)
# the coverage engine
self.director = CoverageDirector(self.metadata, self.core.palette)
# the coverage painter
self.painter = CoveragePainter(self, self.director, self.core.palette)
# the coverage overview widget
self.coverage_overview = None
# the directory to start the coverage file dialog in
self._last_directory = None
@property
def palette(self):
return self.core.palette
def start(self):
"""
One-time activation a Lighthouse context and its subsystems.
"""
if self._started:
return
self.core.palette.warmup()
self.metadata.start()
self.director.start()
self.painter.start()
# TODO/BINJA remove this ASAP, or find a better workaround... I hate having this here
if disassembler.NAME == "BINJA":
disassembler.hide_dockable("Feature Map")
self._started = True
def terminate(self):
"""
Spin down any session subsystems before the session is deleted.
"""
if not self._started:
return
self.painter.terminate()
self.director.terminate()
self.metadata.terminate()
def select_coverage_files(self):
"""
Prompt a file selection dialog, returning file selections.
NOTE: This saves & reuses the last known directory for subsequent uses.
"""
if not self._last_directory:
self._last_directory = disassembler[self].get_database_directory()
# create & configure a Qt File Dialog for immediate use
file_dialog = QtWidgets.QFileDialog(
None,
'Open code coverage file',
self._last_directory,
'All Files (*.*)'
)
file_dialog.setFileMode(QtWidgets.QFileDialog.ExistingFiles)
# prompt the user with the file dialog, and await filename(s)
filenames, _ = file_dialog.getOpenFileNames()
#
# remember the last directory we were in (parsed from a selected file)
# for the next time the user comes to load coverage files
#
if filenames:
self._last_directory = os.path.dirname(filenames[0]) + os.sep
# log the captured (selected) filenames from the dialog
logger.debug("Captured filenames from file dialog:")
for name in filenames:
logger.debug(" - %s" % name)
# return the captured filenames
return filenames
@@ -1,9 +1,12 @@
import os
import time
import logging
import weakref
import itertools
import collections
from lighthouse.util import *
from lighthouse.palette import compute_color_on_gradiant
from lighthouse.util.qt import compute_color_on_gradient
from lighthouse.metadata import DatabaseMetadata
logger = logging.getLogger("Lighthouse.Coverage")
@@ -29,8 +32,6 @@ logger = logging.getLogger("Lighthouse.Coverage")
# get updated or refreshed by the user.
#
BADADDR = 0xFFFFFFFFFFFFFFFF
#------------------------------------------------------------------------------
# Database Coverage
#------------------------------------------------------------------------------
@@ -51,6 +52,12 @@ class DatabaseCoverage(object):
# the filepath this coverage data was sourced from
self.filepath = filepath
# the timestamp of the coverage file on disk
try:
self.timestamp = os.path.getmtime(filepath)
except (OSError, TypeError):
self.timestamp = time.time()
#
# this is the coverage mapping's reference to the underlying database
# metadata. it will use this for all its mapping operations.
@@ -89,7 +96,8 @@ class DatabaseCoverage(object):
# the addresses executed in the coverage log
#
self._hitmap = build_hitmap(data)
self._hitmap = collections.Counter(data)
self._imagebase = BADADDR
#
# the coverage hash is a simple hash of the coverage mask (hitmap keys)
@@ -131,9 +139,7 @@ class DatabaseCoverage(object):
# initially, all loaded coverage data is marked as unmapped
#
self._unmapped_data = set(self._hitmap.keys())
self._unmapped_data.add(BADADDR)
self._misaligned_data = set()
self.unmapped_addresses = set(self._hitmap.keys())
#
# at runtime, the map_coverage() member function of this class is
@@ -158,6 +164,13 @@ class DatabaseCoverage(object):
self.functions = {}
self.instruction_percent = 0.0
# blocks that have not been fully executed (eg, crash / exception)
self.partial_nodes = set()
self.partial_instructions = set()
# addresses that have been executed, but are not in a defined node
self.orphan_addresses = set()
#
# we instantiate a single weakref of ourself (the DatbaseCoverage
# object) such that we can distribute it to the children we create
@@ -180,9 +193,9 @@ class DatabaseCoverage(object):
@property
def coverage(self):
"""
Return the instruction-level coverage bitmap/mask.
Return the coverage (address) bitmap/mask.
"""
return self._hitmap.viewkeys()
return viewkeys(self._hitmap)
@property
def suspicious(self):
@@ -191,6 +204,8 @@ class DatabaseCoverage(object):
"""
bad = 0
total = len(self.nodes)
if not total:
return 0.0
#
# count the number of nodes (basic blocks) that allegedly were executed
@@ -201,7 +216,7 @@ class DatabaseCoverage(object):
# provided coverage data is malformed, or for a different binary
#
for adddress, node_coverage in self.nodes.iteritems():
for adddress, node_coverage in iteritems(self.nodes):
if adddress in node_coverage.executed_instructions:
continue
bad += 1
@@ -230,6 +245,49 @@ class DatabaseCoverage(object):
Install a new databasee metadata object.
"""
self._metadata = weakref.proxy(metadata)
#
# if the underlying database / metadata gets rebased, we will need to
# rebase our coverage data. the 'raw' coverage data stored in the
# hitmap is stored as absolute addresses for performance reasons
#
# here we compute the offset that we will need to rebase the coverage
# data by should a rebase have occurred
#
rebase_offset = self._metadata.imagebase - self._imagebase
#
# if the coverage's imagebase is still BADADDR, that means that this
# coverage object hasn't yet been mapped onto a given metadata cache.
#
# that's fine, we just need to initialize our imagebase which should
# (hopefully!) match the imagebase originally used when baking the
# coverage data into an absolute address form.
#
if self._imagebase == BADADDR:
self._imagebase = self._metadata.imagebase
self._normalize_coverage()
#
# if the imagebase for this coverage exists, then it is susceptible to
# being rebased by a metadata update. if rebase_offset is non-zero,
# this is an indicator that a rebase has occurred.
#
# when a rebase occurs in the metadata, we must also rebase our
# coverage data (stored in the hitmap)
#
elif rebase_offset:
self._hitmap = { (address + rebase_offset): hits for address, hits in iteritems(self._hitmap) }
self._imagebase = self._metadata.imagebase
#
# since the metadata has been updated in one form or another, we need
# to trash our existing coverage mapping, and rebuild it from the data.
#
self.unmap_all()
def refresh(self):
@@ -246,8 +304,18 @@ class DatabaseCoverage(object):
# update the coverage hash incase the hitmap changed
self._update_coverage_hash()
# dump the unmappable coverage data
#self.dump_unmapped()
def refresh_theme(self):
"""
Refresh UI facing elements to reflect the current theme.
Does not require @disassembler.execute_ui decorator as no Qt is touched.
"""
for function in self.functions.values():
function.coverage_color = compute_color_on_gradient(
function.instruction_percent,
self.palette.table_coverage_bad,
self.palette.table_coverage_good
)
def _finalize(self, dirty_nodes, dirty_functions):
"""
@@ -261,14 +329,27 @@ class DatabaseCoverage(object):
"""
Finalize the NodeCoverage objects statistics / data for use.
"""
for node_coverage in dirty_nodes.itervalues():
metadata = self._metadata
for address, node_coverage in iteritems(dirty_nodes):
node_coverage.finalize()
# save off a reference to partially executed nodes
if node_coverage.instructions_executed != metadata.nodes[address].instruction_count:
self.partial_nodes.add(address)
else:
self.partial_nodes.discard(address)
# finalize the set of instructions executed in partially executed nodes
instructions = []
for node_address in self.partial_nodes:
instructions.append(self.nodes[node_address].executed_instructions)
self.partial_instructions = set(itertools.chain.from_iterable(instructions))
def _finalize_functions(self, dirty_functions):
"""
Finalize the FunctionCoverage objects statistics / data for use.
"""
for function_coverage in dirty_functions.itervalues():
for function_coverage in itervalues(dirty_functions):
function_coverage.finalize()
def _finalize_instruction_percent(self):
@@ -277,13 +358,13 @@ class DatabaseCoverage(object):
"""
# sum all the instructions in the database metadata
total = sum(f.instruction_count for f in self._metadata.functions.itervalues())
total = sum(f.instruction_count for f in itervalues(self._metadata.functions))
if not total:
self.instruction_percent = 0.0
return
# sum the unique instructions executed across all functions
executed = sum(f.instructions_executed for f in self.functions.itervalues())
executed = sum(f.instructions_executed for f in itervalues(self.functions))
# save the computed percentage of database instructions executed (0 to 1.0)
self.instruction_percent = float(executed) / total
@@ -298,7 +379,7 @@ class DatabaseCoverage(object):
"""
# add the given runtime data to our data source
for address, hit_count in data.iteritems():
for address, hit_count in iteritems(data):
self._hitmap[address] += hit_count
# do not update other internal structures if requested
@@ -309,7 +390,7 @@ class DatabaseCoverage(object):
self._update_coverage_hash()
# mark these touched addresses as dirty
self._unmapped_data |= data.viewkeys()
self.unmapped_addresses |= viewkeys(data)
def add_addresses(self, addresses, update=True):
"""
@@ -328,7 +409,7 @@ class DatabaseCoverage(object):
self._update_coverage_hash()
# mark these touched addresses as dirty
self._unmapped_data |= set(addresses)
self.unmapped_addresses |= set(addresses)
def subtract_data(self, data):
"""
@@ -336,7 +417,7 @@ class DatabaseCoverage(object):
"""
# subtract the given hitmap from our existing hitmap
for address, hit_count in data.iteritems():
for address, hit_count in iteritems(data):
self._hitmap[address] -= hit_count
#
@@ -379,7 +460,7 @@ class DatabaseCoverage(object):
Update the hash of the coverage mask.
"""
if self._hitmap:
self.coverage_hash = hash(frozenset(self._hitmap.viewkeys()))
self.coverage_hash = hash(frozenset(viewkeys(self._hitmap)))
else:
self.coverage_hash = 0
@@ -387,6 +468,60 @@ class DatabaseCoverage(object):
# Coverage Mapping
#--------------------------------------------------------------------------
def _normalize_coverage(self):
"""
Normalize basic block coverage into instruction coverage.
TODO: It would be interesting if we could do away with this entirely,
working off the original instruction/bb coverage data (hitmap) instead.
"""
coverage_addresses = viewkeys(self._hitmap)
if not coverage_addresses:
return
# bucketize the exploded coverage addresses
instructions = coverage_addresses & self._metadata.instructions
basic_blocks = instructions & viewkeys(self._metadata.nodes)
#
# here we attempt to compute the ratio between basic block addresses,
# and instruction addresses in the incoming coverage data.
#
# this will help us determine if the existing instruction data is
# sufficient, or whether we need to explode/flatten the basic block
# addresses into their respective child instructions
#
block_ratio = len(basic_blocks) / float(len(instructions))
block_trace_confidence = 0.80
logger.debug("Block confidence %f" % block_ratio)
#
# a low basic block to instruction ratio implies the data is probably
# from an instruction trace, or a drcov trace that was exploded from
# (bb_address, size) into its respective addresses
#
if block_ratio < block_trace_confidence:
return
#
# take each basic block address, and explode it into a list of all the
# instruction addresses contained within the basic block as determined
# by the database metadata cache
#
# it is *possible* that this may introduce 'inaccurate' paint should
# the user provide a basic block trace that crashes mid-block. but
# that is not something we can account for in a block trace...
#
for bb_address in basic_blocks:
bb_hits = self._hitmap[bb_address]
for inst_address in self._metadata.nodes[bb_address].instructions:
self._hitmap[inst_address] = bb_hits
logger.debug("Converted basic block trace to instruction trace...")
def _map_coverage(self):
"""
Map loaded coverage data to the underlying database metadata.
@@ -399,10 +534,11 @@ class DatabaseCoverage(object):
"""
Map loaded coverage data to database defined nodes (basic blocks).
"""
db_metadata = self._metadata
dirty_nodes = {}
# the coverage data we will attempt to process in this function
coverage_addresses = collections.deque(sorted(self._unmapped_data))
coverage_addresses = sorted(self.unmapped_addresses)
#
# the loop below is the core of our coverage mapping process.
@@ -420,23 +556,27 @@ class DatabaseCoverage(object):
# speed. please be careful if you wish to modify it...
#
while coverage_addresses:
i, num_addresses = 0, len(coverage_addresses)
while i < num_addresses:
# get the next coverage address to map
address = coverage_addresses.popleft()
address = coverage_addresses[i]
# get the node (basic block) metadata that this address falls in
node_metadata = self._metadata.get_node(address)
node_metadata = db_metadata.get_node(address)
#
# should we fail to locate node metadata for the coverage address
# that we are trying to map, then the address must not fall inside
# of a defined function.
#
# in this case, the coverage address will remain unmapped...
# of a defined function
#
if not node_metadata:
self.orphan_addresses.add(address)
if address in db_metadata.instructions:
self.unmapped_addresses.discard(address)
i += 1
continue
#
@@ -459,8 +599,9 @@ class DatabaseCoverage(object):
node_coverage = NodeCoverage(node_metadata.address, self._weak_self)
self.nodes[node_metadata.address] = node_coverage
# compute the end address of the current basic block
node_end = node_metadata.address + node_metadata.size
# alias for speed, prior to looping
node_start = node_metadata.address
node_end = node_start + node_metadata.size
#
# the loop below is as an inlined fast-path that assumes the next
@@ -474,39 +615,32 @@ class DatabaseCoverage(object):
while 1:
#
# map the hitmap data for the current address (an instruction)
# to this NodeCoverage and mark the instruction as mapped by
# discarding its address from the unmapped data list
# map the hitmap data for the current address if it falls on
# an actual instruction start within the node
#
# if the address falls within an instruction, it will just be
# 'ignored', remaining in the 'unmapped' / invisible data
#
if address in node_metadata.instructions:
node_coverage.executed_instructions[address] = self._hitmap[address]
self._unmapped_data.discard(address)
#
# if the given address allegedly falls within this node's
# address range, but doesn't line up with the known
# instructions, log it as 'misaligned' / suspicious
#
else:
self._misaligned_data.add(address)
self.unmapped_addresses.discard(address)
# get the next address to attempt mapping on
try:
address = coverage_addresses.popleft()
i += 1
address = coverage_addresses[i]
# an IndexError implies there is nothing left to map...
except IndexError:
break;
break
#
# if the next address is not in this node, it's time break out
# of this loop and send it through the full node lookup path
#
if not (node_metadata.address <= address < node_end):
coverage_addresses.appendleft(address)
if not (node_start <= address < node_end):
break
# the node was updated, so save its coverage as dirty
@@ -527,7 +661,7 @@ class DatabaseCoverage(object):
# build or update the function level coverage metadata
#
for node_coverage in dirty_nodes.itervalues():
for node_coverage in itervalues(dirty_nodes):
#
# using a given NodeCoverage object, we retrieve its underlying
@@ -535,28 +669,31 @@ class DatabaseCoverage(object):
# (parent) metadata.
#
function_metadata = self._metadata.nodes[node_coverage.address].function
functions = self._metadata.get_functions_by_node(node_coverage.address)
#
# now we will attempt to retrieve the the FunctionCoverage object
# now we will attempt to retrieve the FunctionCoverage objects
# that we need to parent the given NodeCoverage object to
#
function_coverage = self.functions.get(function_metadata.address, None)
for function_metadata in functions:
function_coverage = self.functions.get(function_metadata.address, None)
#
# if we failed to locate a FunctionCoverage for this node, it means
# that this is the first time we have seen coverage for this
# function. create a new coverage function object and use it now.
#
#
# if we failed to locate the FunctionCoverage for a function
# that references this node, then it is the first time we have
# seen coverage for it.
#
# create a new coverage function object and use it now.
#
if not function_coverage:
function_coverage = FunctionCoverage(function_metadata.address, self._weak_self)
self.functions[function_metadata.address] = function_coverage
if not function_coverage:
function_coverage = FunctionCoverage(function_metadata.address, self._weak_self)
self.functions[function_metadata.address] = function_coverage
# add the NodeCoverage object to its parent FunctionCoverage
function_coverage.mark_node(node_coverage)
dirty_functions[function_metadata.address] = function_coverage
# add the NodeCoverage object to its parent FunctionCoverage
function_coverage.mark_node(node_coverage)
dirty_functions[function_metadata.address] = function_coverage
# done, return a map of FunctionCoverage objects that were modified
return dirty_functions
@@ -565,23 +702,16 @@ class DatabaseCoverage(object):
"""
Unmap all mapped coverage data.
"""
self._unmapped_data = set(self._hitmap.keys())
self._unmapped_data.add(BADADDR)
self._misaligned_data = set()
# clear out the processed / computed coverage data structures
self.nodes = {}
self.functions = {}
self.partial_nodes = set()
self.partial_instructions = set()
self.orphan_addresses = set()
#--------------------------------------------------------------------------
# Debug
#--------------------------------------------------------------------------
def dump_unmapped(self):
"""
Dump the unmapped coverage data.
"""
lmsg("Unmapped Coverage:")
for address in self._unmapped_data:
lmsg(" * 0x%X" % address)
# dump the source coverage data back into an 'unmapped' state
self.unmapped_addresses = set(self._hitmap.keys())
#------------------------------------------------------------------------------
# Function Coverage
@@ -615,7 +745,7 @@ class FunctionCoverage(object):
"""
Return the number of instruction executions in this function.
"""
return sum(x.hits for x in self.nodes.itervalues())
return sum(x.hits for x in itervalues(self.nodes))
@property
def nodes_executed(self):
@@ -629,14 +759,14 @@ class FunctionCoverage(object):
"""
Return the number of unique instructions executed in this function.
"""
return sum(x.instructions_executed for x in self.nodes.itervalues())
return sum(x.instructions_executed for x in itervalues(self.nodes))
@property
def instructions(self):
"""
Return the executed instruction addresses in this function.
"""
return set([ea for node in self.nodes.itervalues() for ea in node.executed_instructions.keys()])
return set([ea for node in itervalues(self.nodes) for ea in node.executed_instructions.keys()])
#--------------------------------------------------------------------------
# Controls
@@ -662,16 +792,16 @@ class FunctionCoverage(object):
float(self.instructions_executed) / function_metadata.instruction_count
# the sum of node executions in this function
node_sum = sum(x.executions for x in self.nodes.itervalues())
node_sum = sum(x.executions for x in itervalues(self.nodes))
# the estimated number of executions this function has experienced
self.executions = float(node_sum) / function_metadata.node_count
# bake colors
self.coverage_color = compute_color_on_gradiant(
self.coverage_color = compute_color_on_gradient(
self.instruction_percent,
self.database.palette.coverage_bad,
self.database.palette.coverage_good
self.database.palette.table_coverage_bad,
self.database.palette.table_coverage_good
)
#------------------------------------------------------------------------------
@@ -687,6 +817,7 @@ class NodeCoverage(object):
self.database = database
self.address = node_address
self.executed_instructions = {}
self.instructions_executed = 0
#--------------------------------------------------------------------------
# Properties
@@ -697,14 +828,7 @@ class NodeCoverage(object):
"""
Return the number of instruction executions in this node.
"""
return sum(self.executed_instructions.itervalues())
@property
def instructions_executed(self):
"""
Return the number of unique instructions executed in this node.
"""
return len(self.executed_instructions)
return sum(itervalues(self.executed_instructions))
#--------------------------------------------------------------------------
# Controls
@@ -718,3 +842,6 @@ class NodeCoverage(object):
# the estimated number of executions this node has experienced.
self.executions = float(self.hits) / node_metadata.instruction_count
# the number of unique instructions executed
self.instructions_executed = len(self.executed_instructions)
@@ -1,16 +1,21 @@
import os
import time
import Queue
import string
import logging
import threading
import collections
from lighthouse.util import lmsg
from lighthouse.util.misc import *
from lighthouse.util.qt import await_future, await_lock, color_text
from lighthouse.util.debug import *
from lighthouse.util.python import *
from lighthouse.util.qt import await_future, await_lock, flush_qt_events
from lighthouse.util.disassembler import disassembler
from lighthouse.ui import ModuleSelector
from lighthouse.reader import CoverageReader
from lighthouse.metadata import DatabaseMetadata, metadata_progress
from lighthouse.coverage import DatabaseCoverage
from lighthouse.exceptions import *
from lighthouse.composer.parser import *
logger = logging.getLogger("Lighthouse.Director")
@@ -46,27 +51,39 @@ class CoverageDirector(object):
between any number of coverage files.
"""
ERROR_COVERAGE_ABSENT = 1
ERROR_COVERAGE_SUSPICIOUS = 2
def __init__(self, metadata, palette):
def __init__(self, palette):
# the database metadata cache
self.metadata = metadata
# the plugin color palette
self._palette = palette
# the central database metadata cache
self.metadata = DatabaseMetadata()
self.palette = palette
#----------------------------------------------------------------------
# Coverage
#----------------------------------------------------------------------
# the coverage file parser
self.reader = CoverageReader()
self._target_whitelist = []
self.suppressed_errors = set()
# the name of the active coverage
self.coverage_name = NEW_COMPOSITION
# a map of loaded or composed database coverages
self._database_coverage = collections.OrderedDict()
#
# the owners map is used in block/coverage blame operations. it
# contains the mapping of node_address --> [ coverage filepaths ]
#
# given any node (basic block) address, we can use this mapping to do
# a reverse lookup to find which loaded coverage sets hit the block.
#
self.owners = collections.defaultdict(set)
#
# the director automatically maintains / generates a few coverage sets
# of its own. these are not directly modifiable by the user, but may
@@ -164,7 +181,7 @@ class CoverageDirector(object):
# to handle these computation requests.
#
self._ast_queue = Queue.Queue()
self._ast_queue = queue.Queue()
self._composition_lock = threading.Lock()
self._composition_cache = CompositionCache()
@@ -172,7 +189,6 @@ class CoverageDirector(object):
target=self._async_evaluate_ast,
name="EvaluateAST"
)
self._composition_worker.start()
#----------------------------------------------------------------------
# Callbacks
@@ -194,20 +210,37 @@ class CoverageDirector(object):
self._coverage_created_callbacks = []
self._coverage_deleted_callbacks = []
# metadata callbacks
self._metadata_modified_callbacks = []
# director callbacks
self._refreshed_callbacks = []
#--------------------------------------------------------------------------
# Subsystem Lifetime
#--------------------------------------------------------------------------
def start(self):
"""
Start the metadata subsystem.
"""
self._composition_worker.start()
def terminate(self):
"""
Cleanup & terminate the director.
"""
# stop the composition worker
self._ast_queue.put(None)
self._composition_worker.join()
try:
self._composition_worker.join()
except RuntimeError:
pass
# spin down the live metadata object
self.metadata.terminate()
# best effort to free up resources & improve interpreter spindown
del self._special_coverage
del self._database_coverage
del self._coverage_switched_callbacks
del self._coverage_modified_callbacks
del self._coverage_created_callbacks
del self._coverage_deleted_callbacks
del self._composition_cache
#--------------------------------------------------------------------------
# Properties
@@ -232,14 +265,14 @@ class CoverageDirector(object):
"""
Return the list or loaded / composed database coverage names.
"""
return self._database_coverage.keys()
return list(self._database_coverage)
@property
def special_names(self):
"""
Return the list of special (director maintained) coverage names.
"""
return self._special_coverage.keys()
return list(self._special_coverage)
@property
def all_names(self):
@@ -285,8 +318,6 @@ class CoverageDirector(object):
def _notify_coverage_created(self):
"""
Notify listeners of a coverage creation event.
TODO/FUTURE: send list of names created?
"""
notify_callback(self._coverage_created_callbacks)
@@ -299,22 +330,27 @@ class CoverageDirector(object):
def _notify_coverage_deleted(self):
"""
Notify listeners of a coverage deletion event.
TODO/FUTURE: send list of names deleted?
"""
notify_callback(self._coverage_deleted_callbacks)
def metadata_modified(self, callback):
def refreshed(self, callback):
"""
Subscribe a callback for metadata modification events.
Subscribe a callback for director refresh events.
"""
register_callback(self._metadata_modified_callbacks, callback)
return register_callback(self._refreshed_callbacks, callback)
def _notify_metadata_modified(self):
def _notify_refreshed(self):
"""
Notify listeners of a metadata modification event.
Notify listeners of a director refresh event.
"""
notify_callback(self._metadata_modified_callbacks)
notify_callback(self._refreshed_callbacks)
def unregister_refreshed(self, callback_ref):
"""
Unsubscribe a callback reference from director refresh events.
"""
unregister_callback(self._refreshed_callbacks, callback_ref)
#----------------------------------------------------------------------
# Batch Loading
@@ -339,176 +375,416 @@ class CoverageDirector(object):
self._aggregation_suspended = True
#----------------------------------------------------------------------
# Coverage Creation
# Coverage Loading
#----------------------------------------------------------------------
def load_coverage_batch(self, filepaths, batch_name, progress_callback=logger.debug):
"""
Create a new database coverage mapping from a list of coverage files.
Returns a tuple of (coverage, errors)
"""
errors = collections.defaultdict(list)
aggregate_addresses = set()
# unsupress NO_COVERAGE_ERROR per-load, instead of per-session
self.suppressed_errors.discard(CoverageMissingError)
start = time.time()
#----------------------------------------------------------------------
for i, filepath in enumerate(filepaths, 1):
logger.debug("-"*50)
progress_callback("Aggregating batch data %u/%u" % (i, len(filepaths)))
# attempt to load coverage data from disk
try:
coverage_file = self.reader.open(filepath)
coverage_addresses = self._extract_coverage_data(coverage_file)
# save and suppress warnings generated from loading coverage files
except CoverageParsingError as e:
errors[CoverageParsingError].append(e)
continue
# ensure some data was actually extracted from the log
if not coverage_addresses:
errors[CoverageMissingError].append(CoverageMissingError(filepath))
continue
# save the attribution data for this coverage data
for address in coverage_addresses:
if address in self.metadata.nodes:
self.owners[address].add(filepath)
# aggregate all coverage data into a single set of addresses
aggregate_addresses.update(coverage_addresses)
if not aggregate_addresses:
return (None, errors)
# save the batched coverage data to the director
coverage = self.create_coverage(batch_name, aggregate_addresses)
# evaluate coverage
if not coverage.nodes:
errors[CoverageMappingAbsent].append(CoverageMappingAbsent(coverage))
elif coverage.suspicious:
errors[CoverageMappingSuspicious].append(CoverageMappingSuspicious(coverage))
#----------------------------------------------------------------------
end = time.time()
logger.debug("Batch loading took %f seconds" % (end-start))
# return the created coverage name
return (coverage, errors)
def load_coverage_files(self, filepaths, progress_callback=logger.debug):
"""
Create new database coverage mappings from a list of coverage files.
Returns a tuple of (created_coverage, errors)
"""
errors = collections.defaultdict(list)
all_coverage = []
# unsupress NO_COVERAGE_ERROR per-load, instead of per-session
self.suppressed_errors.discard(CoverageMissingError)
start = time.time()
#----------------------------------------------------------------------
#
# stop the director's aggregate set from recomputing after each new
# coverage mapping is created. instead, we want to wait till *all* new
# files have been loaded and mapped, computing the new aggregate only
# at very end. this is far more performant.
#
self.suspend_aggregation()
#
# loop through the list of filepaths we have been given and begin
# the process of loading the coverage data from disk, and normalizing
# it for the director to consume
#
for i, filepath in enumerate(filepaths, 1):
logger.debug("-"*50)
progress_callback("Loading coverage %u/%u" % (i, len(filepaths)))
# attempt to load coverage data from disk
try:
coverage_file = self.reader.open(filepath)
coverage_addresses = self._extract_coverage_data(coverage_file)
# save and suppress warnings generated from loading coverage files
except CoverageParsingError as e:
errors[CoverageParsingError].append(e)
continue
# ensure some data was actually extracted from the log
if not coverage_addresses:
errors[CoverageMissingError].append(CoverageMissingError(filepath))
continue
#
# request a name for the new coverage mapping that the director will
# generate from the loaded coverage data
#
coverage_name = self._suggest_coverage_name(filepath)
coverage = self.create_coverage(coverage_name, coverage_addresses, filepath)
# save the attribution data for this coverage data
for address in coverage.data:
if address in self.metadata.nodes: # TODO/UNMAPPED: support right click unmapped addrs
self.owners[address].add(filepath)
# evaluate coverage
if not coverage.nodes:
errors[CoverageMappingAbsent].append(CoverageMappingAbsent(coverage))
elif coverage.suspicious:
errors[CoverageMappingSuspicious].append(CoverageMappingSuspicious(coverage))
# add the newly created coverage to the list of coverage to be returned
all_coverage.append(coverage)
#
# resume the director's aggregation service, triggering an update to
# recompute the aggregate set with the newly loaded coverage
#
progress_callback("Recomputing coverage aggregate...")
self.resume_aggregation()
#----------------------------------------------------------------------
end = time.time()
logger.debug("File loading took %f seconds" % (end-start))
# all done
return (all_coverage, errors)
def _extract_coverage_data(self, coverage_file):
"""
Internal routine to extract relevant coverage data from a CoverageFile.
"""
database_target = self.metadata.filename
target_names = [database_target] + self._target_whitelist
#
# inspect the coverage file and extract the module name that seems to
# match the executable loaded by the disassembler (fuzzy lookup) or
# otherwise aliased by the user through the fallback dialog
#
for name in target_names:
module_name = self._find_fuzzy_name(coverage_file, name)
if module_name:
break
#
# if the fuzzy name lookup failed and there are named modules in the
# coverage file, then we will show them to the user and see if they
# can pick out a matching module to load coverage from
#
if not module_name and coverage_file.modules:
#
# earlier in this load, the user opted to ignore future attempts
# to alias or select coverage data. this is useful when trying to
# load a batch of coverage files, where some coverage files
# contain data, but none relevant to this database.
#
if CoverageMissingError in self.suppressed_errors:
return []
#
# show the module selection dialog to the user, and wait for them
# to select something, or close the dialog
#
dialog = ModuleSelector(database_target, coverage_file.modules, coverage_file.filepath)
result = dialog.exec_()
# check if the user opted to ignore future warnings for missing coverage
if dialog.ignore_missing:
self.suppressed_errors.add(CoverageMissingError)
#
# if the user closed the dialog without selecting a name, there's
# nothing we can do for them. return an empty set of coverage data
#
if not result:
return []
# the user selected a module name! use that to extract coverage
module_name = dialog.selected_name
# the user opted to save the selected name as an 'alias'
if dialog.remember_alias:
self._target_whitelist.append(module_name)
#
# (module, offset, size) style logs (eg, drcov)
#
imagebase = self.metadata.imagebase
try:
coverage_blocks = coverage_file.get_offset_blocks(module_name)
coverage_addresses = [imagebase+offset for bb_start, bb_len in coverage_blocks for offset in xrange(bb_start, bb_start+bb_len)]
return coverage_addresses
except NotImplementedError:
pass
#
# (module, offset) style logs (eg, mod+off)
#
try:
coverage_offsets = coverage_file.get_offsets(module_name)
coverage_addresses = [imagebase+offset for offset in coverage_offsets]
return coverage_addresses
except NotImplementedError:
pass
#
# (absolute address) style log (eg, instruction/bb trace)
#
try:
coverage_addresses = coverage_file.get_addresses(module_name)
return coverage_addresses
except NotImplementedError:
pass
# well, this one is probably the fault of the CoverageFile author...
raise NotImplementedError("Incomplete CoverageFile implementation")
def _suggest_coverage_name(self, filepath):
"""
Return a suggested coverage name for the given filepath.
"""
coverage_name = os.path.basename(filepath)
coverage = self.get_coverage(coverage_name)
# no internal conflict, the filename is a unique enough coverage name
if not coverage:
return coverage_name
#
# if there is an existing coverage mapping under this name, odds are
# that the user is re-loading the same coverage file in which case the
# director will overwrite the old DatabaseCoverage object.
#
# however, we have to be careful for the case where the user loads a
# coverage file from a different directory under the same name
#
# e.g:
# - C:\coverage\foo.log
# - C:\coverage\testing\foo.log
#
# in these cases, we will append a suffix to the new coverage file
#
# assign a suffix to the coverage_name in the event of a collision
if coverage.filepath != filepath:
# find a suitable suffix
for i in xrange(2, 1000000):
new_name = "%s_%u" % (coverage_name, i)
if not self.get_coverage(new_name):
break
# save the suffixed name to the return value
coverage_name = new_name
# return the suggested coverage name for the given filepath
return coverage_name
def _find_fuzzy_name(self, coverage_file, target_name):
"""
Return the closest matching module name in the given coverage file.
"""
target_name = target_name.lower()
#
# 0. Pre-process module names, strip filepath if present
#
clean_module_names = {}
for module_name_raw in coverage_file.modules:
# trim 'path' from a 'module name' entry... if present (uncommon)
module_name = os.path.basename(module_name_raw)
#
# if this triggers, it's probably because the coverage file is
# using full filepaths for 'module names', and that there was
# two unique filepaths with the same module name, eg:
#
# - C:\foo.dll
# - C:\bar\foo.dll
#
# this should be super rare, but we'll just revert to using the
# full / unprocessed paths and bail...
#
if module_name in clean_module_names:
clean_module_names = {name: name for name in coverage_file.modules}
break
clean_module_names[module_name] = module_name_raw
#
# 1. exact, case-insensitive filename matching
#
for module_name in clean_module_names:
if target_name == module_name.lower():
return clean_module_names[module_name]
#
# 2. exact, case-insensitive filename matching
#
for module_name in clean_module_names:
if target_name == module_name.lower():
return clean_module_names[module_name]
#
# 2. cleave the extension from the target module name (the source)
# and try again to see if matches anything in the coverage file
#
target_name, extension = os.path.splitext(target_name)
for module_name in clean_module_names:
if target_name == module_name.lower():
return clean_module_names[module_name]
# too risky to do fuzzy matching on short names...
if len(target_name) < 6:
return None
#
# 3. try to match *{target_name}*{extension} in module_name, assuming
# target_name is more than 6 characters and there is no other ambiguity
#
possible_names = []
for module_name in clean_module_names:
if target_name in module_name.lower() and extension in module_name.lower():
possible_names.append(clean_module_names[module_name])
# there were no matches on the wildcarding, so we're done
if not possible_names:
return None
#
# if there is multiple potential matches it is too risky to pick one,
# so we are not going to return anything as a viable match
#
if len(possible_names) > 1:
return None
return possible_names[0]
#----------------------------------------------------------------------
# Coverage Management
#----------------------------------------------------------------------
def get_address_coverage(self, address):
"""
Return a list of database coverage objects containing the given address.
"""
found = []
for name, db_coverage in iteritems(self._database_coverage):
if address in db_coverage.coverage:
found.append(db_coverage)
return found
def get_address_file(self, address):
"""
Return a list of coverage filepaths containing the given address.
"""
node = self.metadata.get_node(address)
if not node:
return []
return list(self.owners.get(node.address, []))
def create_coverage(self, coverage_name, coverage_data, coverage_filepath=None):
"""
Create a new database coverage mapping from the given data.
"""
return self.update_coverage(coverage_name, coverage_data, coverage_filepath)
def create_coverage_from_drcov_list(self, drcov_list):
"""
Create a number of database coverage mappings from a list of DrcovData.
Returns a tuple of (created_coverage, errors)
"""
created_coverage = []
errors = []
#
# stop the director's aggregate from updating. this will prevent the
# aggregate from recomputing after each individual mapping is created.
# instead, we will wait till *all* have been created, computing the
# new aggregate at the very end. this is far more performant.
#
self.suspend_aggregation()
#
# loop through the coverage data we been given (drcov_list), and begin
# the normalization process to translate / filter / flatten its blocks
# into a generic format the director can consume (a list of addresses)
#
for i, drcov_data in enumerate(drcov_list, 1):
# keep the user informed about our progress while loading coverage
disassembler.replace_wait_box(
"Normalizing and mapping coverage %u/%u" % (i, len(drcov_list))
)
#
# translate the coverage data's basic block addresses to the
# imagebase of the open database, and flatten the blocks to a
# list of instruction addresses
#
try:
coverage_data = self._normalize_drcov_data(drcov_data)
except ValueError as e:
errors.append((self.ERROR_COVERAGE_ABSENT, drcov_data.filepath))
lmsg("Failed to normalize coverage %s" % drcov_data.filepath)
lmsg("- %s" % e)
continue
#
# before injecting the new coverage data (now a list of instruction
# addresses), we check to see if there is an existing coverage
# object under the same name.
#
# if there is an existing coverage mapping, odds are that the user
# is probably re-loading the same coverage file in which case we
# simply overwrite the old DatabaseCoverage object.
#
# but we have to be careful for the case where the user loads a
# coverage file from a different directory, but under the same name
#
# e.g:
# - C:\coverage\foo.log
# - C:\coverage\testing\foo.log
#
# in these cases, we will append a suffix to the new coverage file
#
coverage_name = os.path.basename(drcov_data.filepath)
coverage = self.get_coverage(coverage_name)
# assign a suffix to the coverage name in the event of a collision
if coverage and coverage.filepath != drcov_data.filepath:
for i in xrange(2,0x100000):
new_name = "%s_%u" % (coverage_name, i)
if not self.get_coverage(new_name):
break
coverage_name = new_name
#
# finally, we can ask the director to create a coverage mapping
# from the data we have pre-processed for it
#
coverage = self.create_coverage(
coverage_name,
coverage_data,
drcov_data.filepath
)
created_coverage.append(coverage_name)
# warn when loaded coverage appears to be poorly mapped (suspicious)
if coverage.suspicious:
errors.append((self.ERROR_COVERAGE_SUSPICIOUS, drcov_data.filepath))
lmsg("Badly mapped coverage %s" % drcov_data.filepath)
#
# resume the director's aggregation service, triggering an update to
# recompute the aggregate with the newly loaded coverage
#
disassembler.replace_wait_box("Recomputing coverage aggregate...")
self.resume_aggregation()
# done
return (created_coverage, errors)
def _normalize_drcov_data(self, drcov_data):
"""
Extract and normalize relevant coverage data from a DrcovData object.
Returns a list of executed instruction addresses for this database.
"""
# extract the coverage relevant to this database (well, the root binary)
root_filename = self.metadata.filename
coverage_blocks = drcov_data.get_blocks_by_module(root_filename)
# rebase the coverage log's basic blocks to the database imagebase
imagebase = self.metadata.imagebase
rebased_blocks = rebase_blocks(imagebase, coverage_blocks)
# coalesce the blocks into larger contiguous blobs
condensed_blocks = coalesce_blocks(rebased_blocks)
# flatten the blobs into individual instruction addresses
return self.metadata.flatten_blocks(condensed_blocks)
def aggregate_drcov_batch(self, drcov_list):
"""
Aggregate a given list of DrcovData into a single coverage mapping.
See create_coverage_from_drcov_list(...) for more verbose comments.
"""
errors = []
# create a new coverage set to manually aggregate data into
coverage = DatabaseCoverage(self._palette)
for i, drcov_data in enumerate(drcov_list, 1):
# keep the user informed about our progress while aggregating
disassembler.replace_wait_box(
"Aggregating batch data %u/%u" % (i, len(drcov_list))
)
# normalize coverage data to the open database
try:
addresses = self._normalize_drcov_data(drcov_data)
except Exception as e:
errors.append((self.ERROR_COVERAGE_ABSENT, drcov_data.filepath))
lmsg("Failed to normalize coverage %s" % drcov_data.filepath)
lmsg("- %s" % e)
continue
# aggregate the addresses into the output coverage mapping
coverage.add_addresses(addresses, False)
# return the created coverage name
return (coverage, errors)
#----------------------------------------------------------------------
# Coverage Management
#----------------------------------------------------------------------
def select_coverage(self, coverage_name):
"""
Activate a loaded coverage mapping by name.
@@ -547,7 +823,7 @@ class CoverageDirector(object):
# create a new database coverage mapping from the given coverage data
new_coverage = DatabaseCoverage(
self._palette,
self.palette,
coverage_name,
coverage_filepath,
coverage_data
@@ -662,7 +938,7 @@ class CoverageDirector(object):
# TODO/FUTURE: check if there's any references to the coverage aggregate?
# assign a new, blank aggregate set
self._special_coverage[AGGREGATE] = DatabaseCoverage(self._palette, AGGREGATE)
self._special_coverage[AGGREGATE] = DatabaseCoverage(self.palette, AGGREGATE)
self._refresh_aggregate() # probably not needed
def get_coverage(self, name):
@@ -682,7 +958,7 @@ class CoverageDirector(object):
# could not locate coverage
return None
def get_coverage_string(self, coverage_name, color=False):
def get_coverage_string(self, coverage_name):
"""
Retrieve a detailed coverage string for the given coverage_name.
"""
@@ -703,23 +979,6 @@ class CoverageDirector(object):
# eg: 'A - 73.45% - drcov.boombox.exe.03820.0000.proc.log'
#
if color:
# color the symbol token like the shell
symbol = color_text(symbol, self._palette.coverage_token)
# low coverage color
if percent < 30.0:
percent_str = color_text(percent_str, self._palette.coverage_bad)
# okay coverage color
elif percent < 60.0:
percent_str = color_text(percent_str, self._palette.coverage_okay)
# good coverage color
else:
percent_str = color_text(percent_str, self._palette.coverage_good)
return "%s - %s%% - %s" % (symbol, percent_str, coverage_name)
#----------------------------------------------------------------------
@@ -847,6 +1106,7 @@ class CoverageDirector(object):
# evaluate the last AST into a coverage set
composite_coverage = self._evaluate_composition(ast)
composite_coverage.name = composite_name
# save the evaluated coverage under the given name
self._commit_coverage(composite_name, composite_coverage)
@@ -914,7 +1174,7 @@ class CoverageDirector(object):
# if the AST is effectively 'null', return a blank coverage set
if isinstance(ast, TokenNull):
return DatabaseCoverage(self._palette)
return DatabaseCoverage(self.palette)
#
# the director's composition evaluation code (this function) is most
@@ -1052,21 +1312,12 @@ class CoverageDirector(object):
# we use the mask to generate a new DatabaseCoverage mapping.
#
new_composition = DatabaseCoverage(self._palette, data=coverage_mask)
new_composition = DatabaseCoverage(self.palette, data=coverage_mask)
# cache & return the newly computed composition
self._composition_cache[composition_hash] = new_composition
return new_composition
#
# if the current AST node is a coverage range, we need to evaluate the
# range expression. this will produce an aggregate coverage set
# described by the start/end of the range (eg, 'A,D')
#
elif isinstance(node, TokenCoverageRange):
return self._evaluate_coverage_range(node)
#
# if the current AST node is a coverage token, we need simply need to
# return its associated DatabaseCoverage.
@@ -1090,27 +1341,6 @@ class CoverageDirector(object):
assert isinstance(coverage_token, TokenCoverageSingle)
return self.get_coverage(self._alias2name[coverage_token.symbol])
def _evaluate_coverage_range(self, range_token):
"""
Evaluate a TokenCoverageRange AST token.
Returns a new aggregate database coverage mapping.
"""
assert isinstance(range_token, TokenCoverageRange)
# initialize output to a null coverage set
output = DatabaseCoverage(self._palette)
# expand 'A,Z' to ['A', 'B', 'C', ... , 'Z']
symbols = [chr(x) for x in range(ord(range_token.symbol_start), ord(range_token.symbol_end) + 1)]
# build a coverage aggregate described by the range of shorthand symbols
for symbol in symbols:
output.add_data(self.get_coverage(self._alias2name[symbol]).data)
# return the computed coverage
return output
#----------------------------------------------------------------------
# Refresh
#----------------------------------------------------------------------
@@ -1119,49 +1349,63 @@ class CoverageDirector(object):
"""
Complete refresh of the director and mapped coverage.
"""
logger.debug("Refreshing the CoverageDirector")
if disassembler[self.metadata.lctx].busy:
disassembler.warning("Cannot refresh Lighthouse while the disassembler is busy...")
return
# (re)build our metadata cache of the underlying database
future = self.refresh_metadata(metadata_progress, True)
await_future(future)
disassembler.show_wait_box("Refreshing Lighthouse...")
self._refresh()
disassembler.hide_wait_box()
# (re)map each set of loaded coverage data to the database
self._refresh_database_coverage()
def refresh_metadata(self, progress_callback=None, force=False):
@catch_errors
def _refresh(self):
"""
Refresh the database metadata cache utilized by the director.
Returns a future (Queue) that will carry the completion message.
Internal refresh routine, wrapped to help catch bugs for now.
"""
#
# if this is the first time the director is going to use / populate
# the database metadata, register the director for notifications of
# metadata modification (this should only happen once)
#
# TODO/FUTURE: this is a little dirty, but it will suffice.
# (re) build our metadata cache of the underlying database
#
if not self.metadata.cached:
self.metadata.function_renamed(self._notify_metadata_modified)
if not is_mainthread():
self.metadata.refresh(metadata_progress)
#
# if the lighthouse has collected metadata previously for this
# disassembler session (eg, it is cached), ignore a request to refresh
# it unless explicitly told to refresh via force=True
# NOTE: optionally, we call the async vesrion here so that we do not pin
# the mainthread for disassemblers that will primarily read from the
# database in a background thread (eg, Binja)
#
# for example, this refresh action may be called from a UI event or
# clicking 'Open Coverage Overview' (eg, the mainthread). if we pin
# the mainthread while doing database reads from a background thread,
# we cannot post UI updates such as progress updates to the user
#
# using an async refresh allows us to 'softly' spin the main (UI)
# thread and get UI updates while the refresh runs
#
if self.metadata.cached and not force:
fake_queue = Queue.Queue()
fake_queue.put(False)
return fake_queue
else:
future = self.metadata.refresh_async(metadata_progress, force=True)
self.metadata.go_synchronous()
await_future(future)
# start the asynchronous metadata refresh
result_queue = self.metadata.refresh(progress_callback=progress_callback)
# (re) map each set of loaded coverage data to the database
if self.coverage_names:
self._refresh_database_coverage()
# return the queue that can be used to block for the async result
return result_queue
# notify of full-refresh
self._notify_refreshed()
def refresh_theme(self):
"""
Refresh UI facing elements to reflect the current theme.
Does not require @disassembler.execute_ui decorator as no Qt is touched.
"""
for coverage in self._database_coverage.values():
coverage.refresh_theme()
for coverage in self._special_coverage.values():
coverage.refresh_theme()
def _refresh_database_coverage(self):
"""
+146
View File
@@ -0,0 +1,146 @@
from lighthouse.util.log import lmsg
from lighthouse.util.misc import iteritems
from lighthouse.util.disassembler import disassembler
#------------------------------------------------------------------------------
# Exception Definitions
#------------------------------------------------------------------------------
class LighthouseError(Exception):
"""
An error generated by Lighthouse.
"""
def __init__(self, *args, **kwargs):
super(LighthouseError, self).__init__(*args, **kwargs)
#------------------------------------------------------------------------------
# Coverage File Exceptions
#------------------------------------------------------------------------------
class CoverageException(LighthouseError):
"""
A class of errors pertaining to loading & mapping coverage files.
"""
name = NotImplementedError
description = NotImplementedError
def __init__(self, message, filepath):
super(CoverageException, self).__init__(message)
self.filepath = filepath
@property
def verbose(self):
return "Error: %s\n\n%s" % (self.name, self.description)
def __str__(self):
return self.message + " '%s'" % self.filepath
class CoverageParsingError(CoverageException):
"""
An error generated by the CoverageReader when all parsers fail.
"""
name = "PARSE_FAILURE"
description = \
"Failed to parse one or more of the selected coverage files!\n\n" \
" Possible reasons:\n" \
" - You selected a file that was *not* a coverage file.\n" \
" - The selected coverage file is malformed or unreadable.\n" \
" - A suitable parser for the coverage file is not installed.\n\n" \
"Please see the disassembler console for more info..."
def __init__(self, filepath, tracebacks):
super(CoverageParsingError, self).__init__("Failed to parse coverage file", filepath)
self.tracebacks = tracebacks
class CoverageMissingError(CoverageException):
"""
An error generated when no data was extracted from a CoverageFile.
"""
name = "NO_COVERAGE_ERROR"
description = \
"No usable coverage data was extracted from one of the selected files.\n\n" \
" Possible reasons:\n" \
" - You selected a coverage file for the wrong binary.\n" \
" - The name of the executable file used to generate this database\n" \
" is different than the one you collected coverage against.\n" \
" - Your DBI failed to collect any coverage for this binary.\n\n" \
"Please see the disassembler console for more info..."
def __init__(self, filepath):
super(CoverageMissingError, self).__init__("No coverage extracted from file", filepath)
class CoverageMappingAbsent(CoverageException):
"""
A warning generated when coverage data cannot be mapped.
"""
name = "NO_COVERAGE_MAPPED"
description = \
"One or more of the loaded coverage files has no visibly mapped data.\n\n" \
" Possible reasons:\n" \
" - The loaded coverage data does not fall within defined functions.\n" \
" - You loaded an absolute address trace with a different imagebase.\n" \
" - The coverage data might be corrupt or malformed.\n\n" \
"Please see the disassembler console for more info..."
def __init__(self, coverage):
super(CoverageMappingAbsent, self).__init__("No coverage data could be mapped", coverage.filepath)
self.coverage = coverage
class CoverageMappingSuspicious(CoverageException):
"""
A warning generated when coverage data does not appear to match the database.
"""
name = "BAD_COVERAGE_MAPPING"
description = \
"One or more of the loaded coverage files appears to be badly mapped.\n\n" \
" Possible reasons:\n" \
" - You selected the wrong binary/module to load coverage from.\n" \
" - Your coverage file/data is for a different version of the\n" \
" binary that does not match what the disassembler has open.\n" \
" - You recorded self-modifying code or something with very\n" \
" abnormal control flow (obfuscated code, malware, packers).\n" \
" - The coverage data might be corrupt or malformed.\n\n" \
"This means that any coverage displayed by Lighthouse is PROBABLY\n" \
"WRONG and is not be trusted because the coverage data does not\n" \
"appear to match the disassembled binary."
def __init__(self, coverage):
super(CoverageMappingSuspicious, self).__init__("Coverage data appears badly mapped", coverage.filepath)
self.coverage = coverage
#------------------------------------------------------------------------------
# UI Warnings
#------------------------------------------------------------------------------
def warn_errors(errors, ignore=[]):
"""
Warn the user of any encountered errors with a messagebox.
"""
if not errors:
return
for error_type, error_list in iteritems(errors):
#
# loop through the individual instances/files that caused this error
# and dump the results to the disassembler console...
#
lmsg("-"*50)
lmsg("Files reporting %s:" % error_type.name)
for error in error_list:
lmsg(" - %s" % error.filepath)
# suppress popups for certain errors, if the user has specified such
if error_type in ignore:
continue
#
# popup a more verbose error messagebox for the user to read regarding
# this class of error they encountered
#
disassembler.warning(error.verbose)
# done ...
lmsg("-"*50)
@@ -0,0 +1,263 @@
import ctypes
import logging
from binaryninja import PluginCommand
from binaryninjaui import UIAction, UIActionHandler, Menu
from lighthouse.context import LighthouseContext
from lighthouse.integration.core import LighthouseCore
from lighthouse.util.disassembler import disassembler
logger = logging.getLogger("Lighthouse.Binja.Integration")
#------------------------------------------------------------------------------
# Lighthouse Binja Integration
#------------------------------------------------------------------------------
class LighthouseBinja(LighthouseCore):
"""
Lighthouse UI Integration for Binary Ninja.
"""
def __init__(self):
super(LighthouseBinja, self).__init__()
def get_context(self, dctx, startup=True):
"""
Get the LighthouseContext object for a given database context.
In Binary Ninja, a dctx is a BinaryView (BV).
"""
dctx_id = ctypes.addressof(dctx.handle.contents)
#
# create a new LighthouseContext if this is the first time a context
# has been requested for this BNDB / bv
#
if dctx_id not in self.lighthouse_contexts:
# create a new 'context' representing this BNDB / bv
lctx = LighthouseContext(self, dctx)
if startup:
lctx.start()
# save the created ctx for future calls
self.lighthouse_contexts[dctx_id] = lctx
#
# for binja, we basically *never* want to start the lighthouse ctx
# when it is first created. this is because binja will *immediately*
# create a coverage overview widget for every database when it is
# first opened.
#
# this is annoying, because we don't want to actually start up all
# of the lighthouse threads and subsystems unless the user actually
# starts trying to use lighthouse for their session.
#
# so we initialize the lighthouse context (with start()) on the
# second context request which will go through the else block
# below... any subsequent call to start() is effectively a nop!
#
else:
lctx = self.lighthouse_contexts[dctx_id]
lctx.start()
# return the lighthouse context object for this database ctx / bv
return lctx
def binja_close_context(self, dctx):
"""
Attempt to close / spin-down the LighthouseContext for the given dctx.
In Binary Ninja, a dctx is a BinaryView (BV).
"""
dctx_id = ctypes.addressof(dctx.handle.contents)
# fetch the LighthouseContext for the closing BNDB
try:
lctx = self.lighthouse_contexts.pop(dctx_id)
#
# if lighthouse was not actually used for this BNDB / session, then
# the lookup will fail as there is nothing to spindown
#
except KeyError:
return
# spin down the closing context (stop threads, cleanup qt state, etc)
logger.info("Closing a LighthouseContext...")
lctx.terminate()
#--------------------------------------------------------------------------
# UI Integration (Internal)
#--------------------------------------------------------------------------
#
# TODO / HACK / XXX / V35 / 2021: Some of Binja's UI elements (such as the
# terminal) do not get assigned a BV, even if there is only one open.
#
# this is problematic, because if the user 'clicks' onto the terminal, and
# then tries to execute our UIActions (like 'Load Coverage File'), the
# given 'context.binaryView' will be None
#
# in the meantime, we have to use this workaround that will try to grab
# the 'current' bv from the dock. this is not ideal, but it will suffice.
#
# -----------------
#
# XXX: It's now 2024, Binja's UI / API stack has grown a lot. it's more
# powerful and a bunch of the oddities / hacks lighthouse employed for
# binja may no longer apply. this whole file should probably be revisited
# and re-factored at some point point.. sorry if it's hard to follow
#
def _interactive_load_file(self, context):
dctx = disassembler.binja_get_bv_from_dock()
if not dctx:
disassembler.warning("Lighthouse requires an open BNDB to load coverage.")
return
super(LighthouseBinja, self).interactive_load_file(dctx)
def _interactive_load_batch(self, context):
dctx = disassembler.binja_get_bv_from_dock()
if not dctx:
disassembler.warning("Lighthouse requires an open BNDB to load coverage.")
return
super(LighthouseBinja, self).interactive_load_batch(dctx)
def _open_coverage_xref(self, context):
super(LighthouseBinja, self).open_coverage_xref(context.address, context.binaryView)
def _interactive_coverage_xref(self, context):
if context is None:
return
#
# this is a special case where we check if the ctx exists rather than
# blindly creating a new one. again, this is because binja may call
# this function at random times to decide whether it should display the
# XREF menu option.
#
# but asking whether or not the xref menu option should be shown is not
# a good indication of 'is the user actually using lighthouse' so we
# do not want this to be one that creates lighthouse contexts
#
dctx = context.binaryView
if not dctx:
return
dctx_id = ctypes.addressof(dctx.handle.contents)
lctx = self.lighthouse_contexts.get(dctx_id, None)
if not lctx:
return
#
# is there even any coverage loaded into lighthouse? if not, the user
# probably isn't even using it. so don't bother showing the xref action
#
if not lctx.director.coverage_names:
return
if context.view is None:
return
view = context.view
context_menu = view.contextMenu()
#
# Create a new, temporary Coverage Xref action to inject into the
# right click context menu that is being shown...
#
action = "Coverage Xref"
UIAction.registerAction(action)
action_handler = view.actionHandler()
action_handler.bindAction(action, UIAction(self._open_coverage_xref))
context_menu.addAction(action, "Plugins")
def _is_xref_valid(self, dctx, addr):
#
# this is a special case where we check if the ctx exists rather than
# blindly creating a new one. again, this is because binja may call
# this function at random times to decide whether it should display the
# XREF menu option.
#
# but asking whether or not the xref menu option should be shown is not
# a good indidication of 'is the user actually using lighthouse' so we
# do not want this to be one that creates lighthouse contexts
#
dctx_id = ctypes.addressof(dctx.handle.contents)
lctx = self.lighthouse_contexts.get(dctx_id, None)
if not lctx:
return False
# return True if there appears to be coverage loaded...
return bool(lctx.director.coverage_names)
def _open_coverage_overview(self, context):
dctx = disassembler.binja_get_bv_from_dock()
if not dctx:
disassembler.warning("Lighthouse requires an open BNDB to open the overview.")
return
super(LighthouseBinja, self).open_coverage_overview(dctx)
def _stub(self, context):
# XXX: This was added as a last minute bodge prior to releasing v0.9.3,
# it fixes a crash-on-close that was manifesting on binja macOS, when
# using a lambda instead of a concrete function/stub like this.
return None
#--------------------------------------------------------------------------
# Binja Actions
#--------------------------------------------------------------------------
ACTION_LOAD_FILE = "Lighthouse\\Load code coverage file..."
ACTION_LOAD_BATCH = "Lighthouse\\Load code coverage batch..."
ACTION_COVERAGE_XREF = "Lighthouse\\Coverage Xref"
ACTION_COVERAGE_OVERVIEW = "Lighthouse\\Open Coverage Overview"
def _install_load_file(self):
action = self.ACTION_LOAD_FILE
UIAction.registerAction(action)
UIActionHandler.globalActions().bindAction(action, UIAction(self._interactive_load_file))
Menu.mainMenu("Plugins").addAction(action, "Loading", 0)
logger.info("Installed the 'Code coverage file' menu entry")
def _install_load_batch(self):
action = self.ACTION_LOAD_BATCH
UIAction.registerAction(action)
UIActionHandler.globalActions().bindAction(action, UIAction(self._interactive_load_batch))
Menu.mainMenu("Plugins").addAction(action, "Loading", 1)
logger.info("Installed the 'Code coverage batch' menu entry")
def _install_open_coverage_xref(self):
action = self.ACTION_COVERAGE_XREF
UIAction.registerAction(action)
UIActionHandler.globalActions().bindAction(action, UIAction(self._stub, self._interactive_coverage_xref))
Menu.mainMenu("Plugins").addAction(action, "Loading", 2)
# NOTE/V35: Binja automatically creates View --> Show Coverage Overview
def _install_open_coverage_overview(self):
action = self.ACTION_COVERAGE_OVERVIEW
UIAction.registerAction(action)
UIActionHandler.globalActions().bindAction(action, UIAction(self._open_coverage_overview))
Menu.mainMenu("Plugins").addAction(action, "Windows", 0)
logger.info("Installed the 'Open Coverage Overview' menu entry")
# NOTE/V35: Binja doesn't really 'unload' plugins, so whatever...
def _uninstall_load_file(self):
pass
def _uninstall_load_batch(self):
pass
def _uninstall_open_coverage_xref(self):
pass
def _uninstall_open_coverage_overview(self):
pass
@@ -1,7 +1,7 @@
import logging
from lighthouse.util.log import lmsg
from lighthouse.binja_integration import LighthouseBinja
from lighthouse.integration.binja_integration import LighthouseBinja
logger = logging.getLogger("Lighthouse.Binja.Loader")
@@ -23,8 +23,6 @@ logger = logging.getLogger("Lighthouse.Binja.Loader")
# when Binary Ninja is starting up. As such, this is our only opportunity
# to load & integrate Lighthouse.
#
# TODO/V35: it would be nice load/unload plugins with BNDB's like IDA
#
try:
lighthouse = LighthouseBinja()
@@ -32,4 +30,3 @@ try:
except Exception as e:
lmsg("Failed to initialize Lighthouse")
logger.exception("Exception details:")
+412
View File
@@ -0,0 +1,412 @@
import abc
import logging
import lighthouse
from lighthouse.util import lmsg
from lighthouse.util.qt import *
from lighthouse.util.update import check_for_update
from lighthouse.util.disassembler import disassembler
from lighthouse.ui import *
from lighthouse.metadata import metadata_progress
from lighthouse.exceptions import *
logger = logging.getLogger("Lighthouse.Core")
#------------------------------------------------------------------------------
# Lighthouse Plugin Core
#------------------------------------------------------------------------------
class LighthouseCore(object):
__metaclass__ = abc.ABCMeta
#--------------------------------------------------------------------------
# Plugin Metadata
#--------------------------------------------------------------------------
PLUGIN_VERSION = "0.9.4-DEV"
AUTHORS = "Markus Gaasedelen"
DATE = "2024"
#--------------------------------------------------------------------------
# Initialization
#--------------------------------------------------------------------------
def load(self):
"""
Load the plugin, and integrate its UI into the disassembler.
"""
self._update_checked = False
self.lighthouse_contexts = {}
# the plugin color palette
self.palette = LighthousePalette()
self.palette.theme_changed(self.refresh_theme)
def create_coverage_overview(name, parent, dctx):
lctx = self.get_context(dctx, startup=False)
widget = disassembler.create_dockable_widget(parent, name)
overview = CoverageOverview(lctx, widget)
return widget
# the coverage overview widget
disassembler.register_dockable("Coverage Overview", create_coverage_overview)
# install disassembler UI
self._install_ui()
# install entry point for headless / terminal access...
lighthouse.get_context = self.get_context
# plugin loaded successfully, print the plugin banner
self.print_banner()
logger.info("Successfully loaded plugin")
def unload(self):
"""
Unload the plugin, and remove any UI integrations.
"""
self._uninstall_ui()
# remove headless entry point
lighthouse.get_context = lambda x: None
# spin down any active contexts (stop threads, cleanup qt state, etc)
for lctx in self.lighthouse_contexts.values():
lctx.terminate()
logger.info("-"*75)
logger.info("Plugin terminated")
def print_banner(self):
"""
Print the plugin banner.
"""
# build the main banner title
banner_params = (self.PLUGIN_VERSION, self.AUTHORS, self.DATE)
banner_title = "v%s - (c) %s - %s" % banner_params
# print plugin banner
lmsg("Loaded %s" % banner_title)
#--------------------------------------------------------------------------
# Disassembler / Database Context Selector
#--------------------------------------------------------------------------
@abc.abstractmethod
def get_context(self, dctx, startup=True):
"""
Get the LighthouseContext object for a given database context.
"""
pass
#--------------------------------------------------------------------------
# UI Integration (Internal)
#--------------------------------------------------------------------------
def _install_ui(self):
"""
Initialize & integrate all plugin UI elements.
"""
self._install_load_file()
self._install_load_batch()
self._install_open_coverage_xref()
self._install_open_coverage_overview()
def _uninstall_ui(self):
"""
Cleanup & remove all plugin UI integrations.
"""
self._uninstall_open_coverage_overview()
self._uninstall_open_coverage_xref()
self._uninstall_load_batch()
self._uninstall_load_file()
@abc.abstractmethod
def _install_load_file(self):
"""
Install the 'File->Load->Code coverage file...' menu entry.
"""
pass
@abc.abstractmethod
def _install_load_batch(self):
"""
Install the 'File->Load->Code coverage batch...' menu entry.
"""
pass
@abc.abstractmethod
def _install_open_coverage_xref(self):
"""
Install the right click 'Coverage Xref' context menu entry.
"""
pass
@abc.abstractmethod
def _install_open_coverage_overview(self):
"""
Install the 'View->Open subviews->Coverage Overview' menu entry.
"""
pass
@abc.abstractmethod
def _uninstall_load_file(self):
"""
Remove the 'File->Load file->Code coverage file...' menu entry.
"""
pass
@abc.abstractmethod
def _uninstall_load_batch(self):
"""
Remove the 'File->Load file->Code coverage batch...' menu entry.
"""
pass
@abc.abstractmethod
def _uninstall_open_coverage_xref(self):
"""
Remove the right click 'Coverage Xref' context menu entry.
"""
pass
@abc.abstractmethod
def _uninstall_open_coverage_overview(self):
"""
Remove the 'View->Open subviews->Coverage Overview' menu entry.
"""
pass
#--------------------------------------------------------------------------
# UI Actions (Public)
#--------------------------------------------------------------------------
def refresh_theme(self):
"""
Refresh UI facing elements to reflect the current theme.
"""
for lctx in self.lighthouse_contexts.values():
lctx.director.refresh_theme()
if lctx.coverage_overview:
lctx.coverage_overview.refresh_theme()
lctx.painter.force_repaint()
def open_coverage_overview(self, dctx=None):
"""
Open the dockable 'Coverage Overview' dialog.
"""
lctx = self.get_context(dctx)
# the coverage overview is already open & visible, nothing to do
if lctx.coverage_overview and lctx.coverage_overview.visible:
return
# show the coverage overview
disassembler.show_dockable("Coverage Overview")
# trigger an update check (this should only ever really 'check' once)
self.check_for_update()
def open_coverage_xref(self, address, dctx=None):
"""
Open the 'Coverage Xref' dialog for a given address.
"""
lctx = self.get_context(dctx)
# show the coverage xref dialog
dialog = CoverageXref(lctx.director, address)
if not dialog.exec_():
return
# activate the user selected xref (if one was double clicked)
if dialog.selected_coverage:
lctx.director.select_coverage(dialog.selected_coverage)
return
# load a coverage file from disk
disassembler.show_wait_box("Loading coverage from disk...")
created_coverage, errors = lctx.director.load_coverage_files(
[dialog.selected_filepath],
disassembler.replace_wait_box
)
if not created_coverage:
lmsg("No coverage files could be loaded...")
disassembler.hide_wait_box()
warn_errors(errors)
return
disassembler.replace_wait_box("Selecting coverage...")
lctx.director.select_coverage(created_coverage[0].name)
disassembler.hide_wait_box()
def interactive_load_batch(self, dctx=None):
"""
Perform the user-interactive loading of a coverage batch.
"""
lctx = self.get_context(dctx)
#
# kick off an asynchronous metadata refresh. this will run in the
# background while the user is selecting which coverage files to load
#
future = lctx.metadata.refresh_async(progress_callback=metadata_progress)
#
# we will now prompt the user with an interactive file dialog so they
# can select the coverage files they would like to load from disk
#
filepaths = lctx.select_coverage_files()
if not filepaths:
lctx.director.metadata.abort_refresh()
return
# prompt the user to name the new coverage aggregate
default_name = "BATCH_%s" % lctx.director.peek_shorthand()
ok, batch_name = prompt_string(
"Batch Name:",
"Please enter a name for this coverage",
default_name
)
#
# if user didn't enter a name for the batch (or hit cancel) we should
# abort the loading process...
#
if not (ok and batch_name):
lmsg("User failed to enter a name for the batch coverage...")
lctx.director.metadata.abort_refresh()
return
#
# to begin mapping the loaded coverage data, we require that the
# asynchronous database metadata refresh has completed. if it is
# not done yet, we will block here until it completes.
#
# a progress dialog depicts the work remaining in the refresh
#
disassembler.show_wait_box("Building database metadata...")
lctx.metadata.go_synchronous()
await_future(future)
#
# now that the database metadata is available, we can use the director
# to normalize and condense (aggregate) all the coverage data
#
disassembler.replace_wait_box("Loading coverage from disk...")
batch_coverage, errors = lctx.director.load_coverage_batch(
filepaths,
batch_name,
disassembler.replace_wait_box
)
# if batch creation fails...
if not batch_coverage:
lmsg("Creation of batch '%s' failed..." % batch_name)
disassembler.hide_wait_box()
warn_errors(errors)
return
# select the newly created batch coverage
disassembler.replace_wait_box("Selecting coverage...")
lctx.director.select_coverage(batch_name)
# all done! pop the coverage overview to show the user their results
disassembler.hide_wait_box()
lmsg("Successfully loaded batch %s..." % batch_name)
self.open_coverage_overview(lctx.dctx)
# finally, emit any notable issues that occurred during load
warn_errors(errors, lctx.director.suppressed_errors)
def interactive_load_file(self, dctx=None):
"""
Perform the user-interactive loading of individual coverage files.
"""
lctx = self.get_context(dctx)
#
# kick off an asynchronous metadata refresh. this will run in the
# background while the user is selecting which coverage files to load
#
future = lctx.metadata.refresh_async(progress_callback=metadata_progress)
#
# we will now prompt the user with an interactive file dialog so they
# can select the coverage files they would like to load from disk
#
filenames = lctx.select_coverage_files()
if not filenames:
lctx.metadata.abort_refresh()
return
#
# to begin mapping the loaded coverage data, we require that the
# asynchronous database metadata refresh has completed. if it is
# not done yet, we will block here until it completes.
#
# a progress dialog depicts the work remaining in the refresh
#
disassembler.show_wait_box("Building database metadata...")
lctx.metadata.go_synchronous()
await_future(future)
#
# now that the database metadata is available, we can use the director
# to load and normalize the selected coverage files
#
disassembler.replace_wait_box("Loading coverage from disk...")
created_coverage, errors = lctx.director.load_coverage_files(filenames, disassembler.replace_wait_box)
#
# if the director failed to map any coverage, the user probably
# provided bad files. emit any warnings and bail...
#
if not created_coverage:
lmsg("No coverage files could be loaded...")
disassembler.hide_wait_box()
warn_errors(errors)
return
#
# activate the first of the newly loaded coverage file(s). this is the
# one that will be visible in the coverage overview once opened
#
disassembler.replace_wait_box("Selecting coverage...")
lctx.director.select_coverage(created_coverage[0].name)
# all done! pop the coverage overview to show the user their results
disassembler.hide_wait_box()
lmsg("Successfully loaded %u coverage file(s)..." % len(created_coverage))
self.open_coverage_overview(lctx.dctx)
# finally, emit any notable issues that occurred during load
warn_errors(errors, lctx.director.suppressed_errors)
def check_for_update(self):
"""
Check if there is an update available for Lighthouse.
"""
if self._update_checked:
return
# wrap the callback (a popup) to ensure it gets called from the UI
callback = disassembler.execute_ui(disassembler.warning)
# kick off the async update check
check_for_update(self.PLUGIN_VERSION, callback)
self._update_checked = True
@@ -2,8 +2,10 @@ import os
import logging
import idaapi
from lighthouse.core import Lighthouse
from lighthouse.context import LighthouseContext
from lighthouse.util.misc import plugin_resource
from lighthouse.integration.core import LighthouseCore
logger = logging.getLogger("Lighthouse.IDA.Integration")
@@ -11,7 +13,7 @@ logger = logging.getLogger("Lighthouse.IDA.Integration")
# Lighthouse IDA Integration
#------------------------------------------------------------------------------
class LighthouseIDA(Lighthouse):
class LighthouseIDA(LighthouseCore):
"""
Lighthouse UI Integration for IDA Pro.
"""
@@ -19,19 +21,52 @@ class LighthouseIDA(Lighthouse):
def __init__(self):
# menu entry icons
self._icon_id_xref = idaapi.BADADDR
self._icon_id_file = idaapi.BADADDR
self._icon_id_batch = idaapi.BADADDR
self._icon_id_overview = idaapi.BADADDR
# IDA ui hooks
self._ui_hooks = UIHooks(self)
# run initialization
super(LighthouseIDA, self).__init__()
def get_context(self, dctx=None, startup=True):
"""
Get the LighthouseContext object for a given database context.
NOTE: since IDA can only have one binary / IDB open at a time, the
dctx (database context) should always be 'None'.
"""
self.palette.warmup()
#
# there should only ever be 'one' disassembler / IDB context at any
# time for IDA. but if one does not exist yet, that means this is the
# first time the user has interacted with Lighthouse for this session
#
if dctx not in self.lighthouse_contexts:
# create a new 'context' representing this IDB
lctx = LighthouseContext(self, dctx)
if startup:
lctx.start()
# save the created ctx for future calls
self.lighthouse_contexts[dctx] = lctx
# return the lighthouse context object for this IDB
return self.lighthouse_contexts[dctx]
#--------------------------------------------------------------------------
# IDA Actions
#--------------------------------------------------------------------------
ACTION_LOAD_FILE = "lighthouse:load_file"
ACTION_LOAD_BATCH = "lighthouse:load_batch"
ACTION_COVERAGE_XREF = "lighthouse:coverage_xref"
ACTION_COVERAGE_OVERVIEW = "lighthouse:coverage_overview"
def _install_load_file(self):
@@ -41,7 +76,7 @@ class LighthouseIDA(Lighthouse):
# create a custom IDA icon
icon_path = plugin_resource(os.path.join("icons", "load.png"))
icon_data = str(open(icon_path, "rb").read())
icon_data = open(icon_path, "rb").read()
self._icon_id_file = idaapi.load_custom_icon(data=icon_data)
# describe a custom IDA UI action
@@ -77,7 +112,7 @@ class LighthouseIDA(Lighthouse):
# create a custom IDA icon
icon_path = plugin_resource(os.path.join("icons", "batch.png"))
icon_data = str(open(icon_path, "rb").read())
icon_data = open(icon_path, "rb").read()
self._icon_id_batch = idaapi.load_custom_icon(data=icon_data)
# describe a custom IDA UI action
@@ -106,6 +141,34 @@ class LighthouseIDA(Lighthouse):
logger.info("Installed the 'Code coverage batch' menu entry")
def _install_open_coverage_xref(self):
"""
Install the right click 'Coverage Xref' context menu entry.
"""
# create a custom IDA icon
icon_path = plugin_resource(os.path.join("icons", "batch.png"))
icon_data = open(icon_path, "rb").read()
self._icon_id_xref = idaapi.load_custom_icon(data=icon_data)
# describe a custom IDA UI action
action_desc = idaapi.action_desc_t(
self.ACTION_COVERAGE_XREF, # The action name
"Xrefs coverage sets...", # The action text
IDACtxEntry(self._pre_open_coverage_xref),# The action handler
None, # Optional: action shortcut
"List coverage sets containing this address", # Optional: tooltip
self._icon_id_xref # Optional: the action icon
)
# register the action with IDA
result = idaapi.register_action(action_desc)
if not result:
RuntimeError("Failed to register coverage_xref action with IDA")
self._ui_hooks.hook()
logger.info("Installed the 'Coverage Xref' menu entry")
def _install_open_coverage_overview(self):
"""
Install the 'View->Open subviews->Coverage Overview' menu entry.
@@ -113,7 +176,7 @@ class LighthouseIDA(Lighthouse):
# create a custom IDA icon
icon_path = plugin_resource(os.path.join("icons", "overview.png"))
icon_data = str(open(icon_path, "rb").read())
icon_data = open(icon_path, "rb").read()
self._icon_id_overview = idaapi.load_custom_icon(data=icon_data)
# describe a custom IDA UI action
@@ -190,6 +253,23 @@ class LighthouseIDA(Lighthouse):
logger.info("Uninstalled the 'Code coverage batch' menu entry")
def _uninstall_open_coverage_xref(self):
"""
Remove the right click 'Coverage Xref' context menu entry.
"""
self._ui_hooks.unhook()
# unregister the action
result = idaapi.unregister_action(self.ACTION_COVERAGE_XREF)
if not result:
return False
# delete the entry's icon
idaapi.free_custom_icon(self._icon_id_xref)
self._icon_id_xref = idaapi.BADADDR
logger.info("Uninstalled the 'Coverage Xref' menu entry")
def _uninstall_open_coverage_overview(self):
"""
Remove the 'View->Open subviews->Coverage Overview' menu entry.
@@ -214,8 +294,37 @@ class LighthouseIDA(Lighthouse):
logger.info("Uninstalled the 'Coverage Overview' menu entry")
#--------------------------------------------------------------------------
# Helpers
#--------------------------------------------------------------------------
def _inject_ctx_actions(self, view, popup, view_type):
"""
Inject context menu entries into IDA's right click menus.
NOTE: This is only being used for coverage xref at this time, but
may host additional actions in the future.
"""
if view_type == idaapi.BWN_DISASMS:
idaapi.attach_action_to_popup(
view,
popup,
self.ACTION_COVERAGE_XREF, # The action ID (see above)
"Xrefs graph from...", # Relative path of where to add the action
idaapi.SETMENU_APP # We want to append the action after ^
)
def _pre_open_coverage_xref(self):
"""
Grab a contextual address before opening the coverage xref dialog.
"""
self.open_coverage_xref(idaapi.get_screen_ea())
#------------------------------------------------------------------------------
# IDA Action Handler Stub
# IDA UI Helpers
#------------------------------------------------------------------------------
class IDACtxEntry(idaapi.action_handler_t):
@@ -239,3 +348,38 @@ class IDACtxEntry(idaapi.action_handler_t):
Ensure the context menu is always available in IDA.
"""
return idaapi.AST_ENABLE_ALWAYS
class UIHooks(idaapi.UI_Hooks):
"""
Hooks for IDA's UI subsystem.
At the moment, we are only using these to inject into IDA's right click
context menus (eg, coverage xrefs)
"""
def __init__(self, integration):
self.integration = integration
super(UIHooks, self).__init__()
def finish_populating_widget_popup(self, widget, popup):
"""
A right click menu is about to be shown. (IDA 7.0+)
"""
#
# if lighthouse hasn't been used yet, there's nothing to do. we also
# don't want this event to trigger the creation of a lighthouse
# context! so we should bail early in this case...
#
if not self.integration.lighthouse_contexts:
return 0
# inject any of lighthouse's right click context menu's into IDA
lctx = self.integration.get_context(None)
if lctx.director.coverage_names:
self.integration._inject_ctx_actions(widget, popup, idaapi.get_widget_type(widget))
# must return 0 for ida...
return 0
@@ -1,8 +1,9 @@
import time
import logging
import idaapi
from lighthouse.util.log import lmsg
from lighthouse.ida_integration import LighthouseIDA
from lighthouse.integration.ida_integration import LighthouseIDA
logger = logging.getLogger("Lighthouse.IDA.Loader")
@@ -65,9 +66,6 @@ class LighthouseIDAPlugin(idaapi.plugin_t):
except Exception as e:
lmsg("Failed to initialize Lighthouse")
logger.exception("Exception details:")
return idaapi.PLUGIN_SKIP
# tell IDA to keep the plugin loaded (everything is okay)
return idaapi.PLUGIN_KEEP
def run(self, arg):
@@ -80,9 +78,17 @@ class LighthouseIDAPlugin(idaapi.plugin_t):
"""
This is called by IDA when it is unloading the plugin.
"""
logger.debug("IDA term started...")
start = time.time()
logger.debug("-"*50)
try:
self._lighthouse.unload()
self._lighthouse = None
except Exception as e:
logger.exception("Failed to cleanly unload Lighthouse from IDA.")
end = time.time()
logger.debug("-"*50)
logger.debug("IDA term done... (%.3f seconds...)" % (end-start))
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,131 @@
import logging
import binaryninja
from binaryninja import HighlightStandardColor
from binaryninja.highlight import HighlightColor
from lighthouse.painting import DatabasePainter
from lighthouse.util.disassembler import disassembler
logger = logging.getLogger("Lighthouse.Painting.Binja")
#------------------------------------------------------------------------------
# Binary Ninja Painter
#------------------------------------------------------------------------------
class BinjaPainter(DatabasePainter):
"""
Asynchronous Binary Ninja database painter.
"""
def __init__(self, lctx, director, palette):
super(BinjaPainter, self).__init__(lctx, director, palette)
#--------------------------------------------------------------------------
# Paint Primitives
#--------------------------------------------------------------------------
#
# NOTE:
# due to the manner in which Binary Ninja implements basic block
# (node) highlighting, there is almost no need to paint individual
# instructions. for now we, will simply make the main instruction
# painting function a no-op's
#
def _paint_instructions(self, instructions):
self._action_complete.set()
def _clear_instructions(self, instructions):
bv = disassembler[self.lctx].bv
state = bv.begin_undo_actions()
for address in instructions:
for func in bv.get_functions_containing(address):
func.set_auto_instr_highlight(address, HighlightStandardColor.NoHighlightColor)
self._painted_partial -= set(instructions)
self._painted_instructions -= set(instructions)
self._action_complete.set()
if hasattr(bv, "forget_undo_actions"):
bv.forget_undo_actions(state)
else:
bv.commit_undo_actions(state)
def _partial_paint(self, bv, instructions, color):
for address in instructions:
for func in bv.get_functions_containing(address):
func.set_auto_instr_highlight(address, color)
self._painted_partial |= set(instructions)
self._painted_instructions |= set(instructions)
def _paint_nodes(self, node_addresses):
bv = disassembler[self.lctx].bv
db_coverage = self.director.coverage
db_metadata = self.director.metadata
state = bv.begin_undo_actions()
r, g, b, _ = self.palette.coverage_paint.getRgb()
color = HighlightColor(red=r, green=g, blue=b)
partial_nodes = set()
for node_address in node_addresses:
node_metadata = db_metadata.nodes.get(node_address, None)
node_coverage = db_coverage.nodes.get(node_address, None)
# read comment in ida_painter.py (self._paint_nodes)
if not (node_coverage and node_metadata):
self._msg_queue.put(self.MSG_ABORT)
node_addresses = node_addresses[:node_addresses.index(node_address)]
break
# special case for nodes that are only partially executed...
if node_coverage.instructions_executed != node_metadata.instruction_count:
partial_nodes.add(node_address)
self._partial_paint(bv, node_coverage.executed_instructions.keys(), color)
continue
for node in bv.get_basic_blocks_starting_at(node_address):
node.highlight = color
self._painted_nodes |= (set(node_addresses) - partial_nodes)
self._action_complete.set()
if hasattr(bv, "forget_undo_actions"):
bv.forget_undo_actions(state)
else:
bv.commit_undo_actions(state)
def _clear_nodes(self, node_addresses):
bv = disassembler[self.lctx].bv
db_metadata = self.director.metadata
state = bv.begin_undo_actions()
for node_address in node_addresses:
node_metadata = db_metadata.nodes.get(node_address, None)
# read comment in ida_painter.py (self._paint_nodes)
if not node_metadata:
self._msg_queue.put(self.MSG_ABORT)
node_addresses = node_addresses[:node_addresses.index(node_address)]
break
for node in bv.get_basic_blocks_starting_at(node_address):
node.highlight = HighlightStandardColor.NoHighlightColor
self._painted_nodes -= set(node_addresses)
self._action_complete.set()
if hasattr(bv, "forget_undo_actions"):
bv.forget_undo_actions(state)
else:
bv.commit_undo_actions(state)
def _refresh_ui(self):
pass
def _cancel_action(self, job):
pass
@@ -1,12 +1,15 @@
import struct
import ctypes
import logging
import functools
import idc
import idaapi
from idaapi import clr_abits, set_abits, netnode, set_node_info
from lighthouse.util import *
from lighthouse.util.disassembler import disassembler
from lighthouse.util.disassembler.ida_api import map_line2citem, map_line2node, lex_citem_indexes
from lighthouse.util.disassembler.ida_api import map_line2citem, map_line2node, lex_citem_indexes, hexrays_available
from lighthouse.painting import DatabasePainter
logger = logging.getLogger("Lighthouse.Painting.IDA")
@@ -39,7 +42,7 @@ logger = logging.getLogger("Lighthouse.Painting.IDA")
# this section of code constitutes some of the most fragile, convoluted,
# and regression prone code in lighthouse. through some miraculous feats
# of engineering, the solution below appears to safely resolve both of
# these problems for downlevel versions (IDA 6.8 --> 7.0)
# these problems for downlevel versions (IDA 6.8 --> 7.x)
#
from lighthouse.util.qt import QtCore
@@ -117,181 +120,231 @@ class IDAPainter(DatabasePainter):
Asynchronous IDA database painter.
"""
def __init__(self, director, palette):
#----------------------------------------------------------------------
# HexRays Hooking
#----------------------------------------------------------------------
#
# we attempt to hook hexrays the *first* time a repaint request is
# made. the assumption being that IDA is fully loaded and if hexrays is
# present, it will definitely be available (for hooking) by this time
#
self._attempted_hook = False
def __init__(self, lctx, director, palette):
super(IDAPainter, self).__init__(lctx, director, palette)
self._streaming_instructions = True
self._idp_hooks = InstructionPaintHooks(director, palette)
self._vduis = {}
# see the MFF_NOWAIT workaround details above
self._signal = ToMainthread()
# continue normal painter initialization
super(IDAPainter, self).__init__(director, palette)
def terminate(self):
def repaint(self):
"""
Paint coverage defined by the current database mappings.
"""
#
# IDA is either closing or simply switching databases... we should try
# to unhook our processor hooks so that artifacts of this painter do
# not carry over to the next IDB / session.
#
# if we don't do this, our current 'IDP' hooks will continue to fire
# once the next IDB is open. we don't want this, because a new painter
# will be spun up an it will install its own instance of hooks...
#
# attempt to hook hexrays *once*
if not self._attempted_hook:
self._init_hexrays_hooks()
self._attempted_hook = True
if self._idp_hooks:
self._idp_hooks.unhook()
self._idp_hooks = None
# execute underlying repaint function
super(IDAPainter, self).repaint()
# spin down the painter as usual
super(IDAPainter, self).terminate()
def _notify_status_changed(self, status):
# enable / disable hook based on the painter being enabled or disabled
if status:
self._idp_hooks.hook()
if hexrays_available():
idaapi.install_hexrays_callback(self._hxe_callback)
else:
self._idp_hooks.unhook()
if hexrays_available():
idaapi.remove_hexrays_callback(self._hxe_callback)
# send the status changed signal...
super(IDAPainter, self)._notify_status_changed(status)
#------------------------------------------------------------------------------
# Paint Actions
#------------------------------------------------------------------------------
#
# NOTE:
# these are 'internal' functions meant only to be used by the painter.
# they are decorated with @execute_paint to force execution into the
# mainthread, where it is safe to paint (in IDA)
#
@execute_paint
def _paint_instructions(self, instructions):
self.paint_instructions(instructions)
"""
Paint instruction level coverage defined by the current database mapping.
NOTE: we now use 'streaming' mode for instructions rather than this.
"""
color = struct.pack("I", self.palette.coverage_paint+1)
for address in instructions:
set_abits(address, 0x40000)
nn = netnode(address)
nn.supset(20, color, 'A')
self._painted_instructions |= set(instructions)
self._action_complete.set()
@execute_paint
def _clear_instructions(self, instructions):
self.clear_instructions(instructions)
self._action_complete.set()
@execute_paint
def _paint_nodes(self, nodes_coverage):
self.paint_nodes(nodes_coverage)
self._action_complete.set()
@execute_paint
def _clear_nodes(self, nodes_metadata):
self.clear_nodes(nodes_metadata)
self._action_complete.set()
@disassembler.execute_ui
def _refresh_ui(self):
idaapi.refresh_idaview_anyway()
def _cancel_action(self, job_id):
idaapi.cancel_exec_request(job_id)
#------------------------------------------------------------------------------
# Paint Actions
#------------------------------------------------------------------------------
def paint_instructions(self, instructions):
"""
Paint instruction level coverage defined by the current database mapping.
"""
for address in instructions:
idaapi.set_item_color(address, self.palette.coverage_paint)
self._painted_instructions |= set(instructions)
def clear_instructions(self, instructions):
"""
Clear paint from the given instructions.
NOTE: we now use 'streaming' mode for instructions rather than this.
"""
for address in instructions:
idaapi.set_item_color(address, idc.DEFCOLOR)
clr_abits(address, 0x40000)
self._painted_instructions -= set(instructions)
self._action_complete.set()
def paint_nodes(self, nodes_coverage):
@execute_paint
def _paint_nodes(self, node_addresses):
"""
Paint node level coverage defined by the current database mappings.
"""
db_metadata = self._director.metadata
db_coverage = self.director.coverage
db_metadata = self.director.metadata
# create a node info object as our vehicle for setting the node color
node_info = idaapi.node_info_t()
# NOTE/COMPAT:
if disassembler.USING_IDA7API:
set_node_info = idaapi.set_node_info
else:
set_node_info = idaapi.set_node_info2
node_info.bg_color = self.palette.coverage_paint
node_flags = idaapi.NIF_BG_COLOR | idaapi.NIF_FRAME_COLOR
#
# loop through every node that we have coverage data for, painting them
# in the IDA graph view as applicable.
#
for node_coverage in nodes_coverage:
node_metadata = db_metadata.nodes[node_coverage.address]
for node_address in node_addresses:
# assign the background color we would like to paint to this node
node_info.bg_color = self.palette.coverage_paint
# retrieve all the necessary structures to paint this node
node_coverage = db_coverage.nodes.get(node_address, None)
functions = db_metadata.get_functions_by_node(node_address)
# do the *actual* painting of a single node instance
set_node_info(
node_metadata.function.address,
node_metadata.id,
node_info,
idaapi.NIF_BG_COLOR | idaapi.NIF_FRAME_COLOR
)
#
# due to the fact that multiple functions may 'share' a node,
# we need to go through and explicitly fetch the node metadata
# from each function when performing a paint.
#
# this is because each function will have a unique node_id in
# the target node_metadata(s)
#
self._painted_nodes.add(node_metadata.address)
node_metadatas = {}
for function in functions:
def clear_nodes(self, nodes_metadata):
# attempt to safely fetch the node metadata from a function
node_metadata = function.nodes.get(node_address, None)
#
# this is possible if function is getting torn down. this is because
# we don't use locks. this just means it is time for us to bail as
# the metadata state is changing and the paint should be canceled
#
if not node_metadata:
node_metadatas = []
break
node_metadatas[function.address] = node_metadata
#
# if we did not get *everything* that we needed, then it is
# possible the database changed, or the coverage set changed...
#
# this is kind of what we get for not using locks :D but that's
# okay, just stop painting here and let the painter sort it out
#
if not (node_coverage and node_metadatas):
self._msg_queue.put(self.MSG_ABORT)
node_addresses = node_addresses[:node_addresses.index(node_address)]
break
# ignore nodes that are only partially executed
if node_coverage.instructions_executed != node_metadata.instruction_count:
continue
# do the *actual* painting o;f a single node instance
for function_address, node_metadata in iteritems(node_metadatas):
set_node_info(
function_address,
node_metadata.id,
node_info,
node_flags
)
self._painted_nodes |= set(node_addresses)
self._action_complete.set()
@execute_paint
def _clear_nodes(self, node_addresses):
"""
Clear paint from the given graph nodes.
"""
db_metadata = self.director.metadata
# create a node info object as our vehicle for resetting the node color
node_info = idaapi.node_info_t()
node_info.bg_color = idc.DEFCOLOR
# NOTE/COMPAT:
if disassembler.USING_IDA7API:
set_node_info = idaapi.set_node_info
else:
set_node_info = idaapi.set_node_info2
node_flags = idaapi.NIF_BG_COLOR | idaapi.NIF_FRAME_COLOR
#
# loop through every node that we have metadata data for, clearing
# their paint (color) in the IDA graph view as applicable.
#
# read self._paint_nodes() comments for more info, the code below
# is very similar, sans the repetitive comments
#
for node_metadata in nodes_metadata:
for node_address in node_addresses:
functions = db_metadata.get_functions_by_node(node_address)
node_metadatas = {}
for function in functions:
node_metadata = function.nodes.get(node_address, None)
if not node_metadata:
node_metadatas = {}
break
node_metadatas[function.address] = node_metadata
# abort if something looks like it changed...
if not node_metadatas:
self._msg_queue.put(self.MSG_ABORT)
node_addresses = node_addresses[:node_addresses.index(node_address)]
break
# do the *actual* painting of a single node instance
set_node_info(
node_metadata.function.address,
node_metadata.id,
node_info,
idaapi.NIF_BG_COLOR | idaapi.NIF_FRAME_COLOR
)
for function_address, node_metadata in iteritems(node_metadatas):
set_node_info(
function_address,
node_metadata.id,
node_info,
node_flags
)
self._painted_nodes.discard(node_metadata.address)
self._painted_nodes -= set(node_addresses)
self._action_complete.set()
@execute_paint
def _refresh_ui(self):
"""
Note that this has been decorated with @execute_paint (vs @execute_ui)
to help avoid deadlocking on exit.
"""
for vdui in self._vduis.values():
if vdui.valid():
vdui.refresh_ctext(False)
idaapi.refresh_idaview_anyway()
def _cancel_action(self, job_id):
if idaapi.IDA_SDK_VERSION < 710:
return
idaapi.cancel_exec_request(job_id)
#------------------------------------------------------------------------------
# Painting - HexRays (Decompilation / Source)
#------------------------------------------------------------------------------
def _init_hexrays_hooks(self):
"""
Install Hex-Rays hooks (when available).
"""
result = False
if idaapi.init_hexrays_plugin():
logger.debug("HexRays present, installing hooks...")
result = idaapi.install_hexrays_callback(self._hxe_callback)
logger.debug("HexRays hooked: %r" % result)
def paint_hexrays(self, cfunc, db_coverage):
"""
Paint decompilation text for the given HexRays Window.
@@ -335,7 +388,7 @@ class IDAPainter(DatabasePainter):
lines_painted = 0
# extract the node addresses that have been hit by our function's mapping data
executed_nodes = set(db_coverage.functions[cfunc.entry_ea].nodes.iterkeys())
executed_nodes = set(viewkeys(db_coverage.functions[cfunc.entry_ea].nodes))
#
# now we loop through every line_number of the decompiled text that claims
@@ -343,7 +396,7 @@ class IDAPainter(DatabasePainter):
# if it contains a node our coverage has marked as executed
#
for line_number, line_nodes in line2node.iteritems():
for line_number, line_nodes in iteritems(line2node):
#
# if there is any intersection of nodes on this line and the coverage
@@ -372,9 +425,6 @@ class IDAPainter(DatabasePainter):
decompilation_text[line_number].bgcolor = self.palette.coverage_paint
lines_painted += 1
# finally, refresh the view
self._refresh_ui()
def _hxe_callback(self, event, *args):
"""
HexRays event handler.
@@ -386,117 +436,39 @@ class IDAPainter(DatabasePainter):
# more code-friendly, readable aliases
vdui = args[0]
cfunc = vdui.cfunc
self._vduis[vdui.view_idx] = vdui
# if there's no coverage data for this function, there's nothing to do
if not cfunc.entry_ea in self._director.coverage.functions:
if not cfunc.entry_ea in self.director.coverage.functions:
return 0
# paint the decompilation text for this function
self.paint_hexrays(cfunc, self._director.coverage)
self.paint_hexrays(cfunc, self.director.coverage)
# stop tracking vdui's if they close...
elif event == idaapi.hxe_close_pseudocode:
vdui = args[0]
self._vduis.pop(vdui.view_idx, None)
return 0
#------------------------------------------------------------------------------
# Priority Painting
#------------------------------------------------------------------------------
#------------------------------------------------------------------------------
# Instruction Paint Streaming (Processor Hooks)
#------------------------------------------------------------------------------
def _priority_paint(self):
"""
Immediately repaint regions of the database visible to the user.
"""
cursor_address = idaapi.get_screen_ea()
class InstructionPaintHooks(idaapi.IDP_Hooks):
"""
Hook IDA's processor callbacks to paint instructions on the fly.
"""
# paint functions around the cursor address
if not self._priority_paint_functions(cursor_address):
return False # a repaint was requested
def __init__(self, director, palette):
super(InstructionPaintHooks, self).__init__()
self.director = director
self.palette = palette
# paint instructions around the cursor address
#if not self._priority_paint_instructions(cursor_address):
# return False # a repaint was requested
# refresh the view
self._refresh_ui()
# successful completion
return True
def _priority_paint_functions(self, target_address):
"""
Paint functions in the immediate vicinity of the given address.
This will paint both the instructions & graph nodes of defined functions.
"""
db_metadata = self._director.metadata
db_coverage = self._director.coverage
# the number of functions before and after the cursor to paint
FUNCTION_BUFFER = 1
# get the function metadata for the function closest to our cursor
function_metadata = db_metadata.get_closest_function(target_address)
if not function_metadata:
return False # a repaint was requested
# select the range of functions around us that we would like to paint
func_num = db_metadata.get_function_index(function_metadata.address)
func_num_start = max(func_num - FUNCTION_BUFFER, 0)
func_num_end = func_num + FUNCTION_BUFFER + 1
# repaint the specified range of functions
for current_num in xrange(func_num_start, func_num_end):
# get the next function to paint
try:
function_metadata = db_metadata.get_function_by_index(current_num)
function_address = function_metadata.address
except IndexError:
continue
# get the function coverage data for the target address
function_coverage = db_coverage.functions.get(function_address, None)
# if there is no function coverage, clear the function
if not function_coverage:
if not self._clear_function(function_address):
return False # a repaint was requested
continue
# there is coverage, so repaint the function
if not self._paint_function(function_address):
return False # a repaint was requested
# paint finished successfully
return True
def _priority_paint_instructions(self, target_address):
"""
Paint instructions in the immediate vicinity of the given address.
"""
db_metadata = self._director.metadata
db_coverage = self._director.coverage
# the number of instruction bytes before and after the cursor to paint
INSTRUCTION_BUFFER = 200
# determine range of instructions to repaint
start_address = max(target_address - INSTRUCTION_BUFFER, 0)
end_address = target_address + INSTRUCTION_BUFFER
instructions = set(db_metadata.get_instructions_slice(start_address, end_address))
# mask only the instructions with coverage data in this region
instructions_coverage = instructions & db_coverage.coverage
#
# clear all instructions in this region, repaint the coverage data
#
# clear instructions
if not self._async_action(self._clear_instructions, instructions):
return False # a repaint was requested
# paint instructions
if not self._async_action(self._paint_instructions, instructions_coverage):
return False # a repaint was requested
# paint finished successfully
return True
def ev_get_bg_color(self, pcolor, ea):
if ea not in self.director.coverage.coverage:
return 0
bgcolor = ctypes.cast(int(pcolor), ctypes.POINTER(ctypes.c_int))
bgcolor[0] = self.palette.coverage_paint
return 1
+671
View File
@@ -0,0 +1,671 @@
import abc
import time
import logging
import threading
from lighthouse.util import *
from lighthouse.util.debug import catch_errors
from lighthouse.coverage import FunctionCoverage
logger = logging.getLogger("Lighthouse.Painting")
class DatabasePainter(object):
"""
An asynchronous disassembler database painting engine.
"""
__metaclass__ = abc.ABCMeta
MSG_ABORT = -1
MSG_TERMINATE = 0
MSG_REPAINT = 1
MSG_FORCE_REPAINT = 2
MSG_CLEAR = 3
MSG_FORCE_CLEAR = 4
MSG_REBASE = 5
def __init__(self, lctx, director, palette):
#----------------------------------------------------------------------
# Misc
#----------------------------------------------------------------------
self.lctx = lctx
self.palette = palette
self.director = director
self._enabled = False
self._started = False
#----------------------------------------------------------------------
# Painted State
#----------------------------------------------------------------------
#
# the coverage painter maintains its own internal record of what
# instruction addresses and graph nodes it has painted.
#
self._imagebase = BADADDR
self._painted_nodes = set()
self._painted_partial = set()
self._painted_instructions = set()
#
# these toggles will let the core painter (this class) know that it
# does not have to order explicit paints of instructions or nodes.
#
# this is because a disassembler-specific painter may be able to hook
# unique callbacks for painting graphs nodes or instructions
# 'on-the-fly' as they are rendered.
#
# these types of paints are ephermal and the most performant, they
# also will not need to be tracked by the painter.
#
self._streaming_nodes = False
self._streaming_instructions = False
#----------------------------------------------------------------------
# Async
#----------------------------------------------------------------------
#
# to communicate with the asynchronous painting thread, we send a
# a message via the thread event to signal a new paint request, and
# use the repaint_requested bool to interrupt a running paint request.
#
self._action_complete = threading.Event()
self._msg_queue = queue.Queue()
self._end_threads = False
#
# asynchronous database painting thread
#
self._painting_worker = threading.Thread(
target=self._async_database_painter,
name="DatabasePainter"
)
#----------------------------------------------------------------------
# Callbacks
#----------------------------------------------------------------------
# painter callbacks
self._status_changed_callbacks = []
# register for cues from the director
self.director.coverage_switched(self.repaint)
self.director.coverage_modified(self.repaint)
self.director.refreshed(self.check_rebase)
def start(self):
"""
Start the painter.
"""
if self._started:
return
# start the painter thread
self._painting_worker.start()
# all done
self._started = True
self.set_enabled(True)
#--------------------------------------------------------------------------
# Status
#--------------------------------------------------------------------------
@property
def enabled(self):
"""
Return the active painting status of the painter.
"""
return self._enabled
def set_enabled(self, enabled):
"""
Enable or disable the painter.
"""
# enabled/disabled status is not changing, ignore...
if enabled == self._enabled:
return
lmsg("%s painting..." % ("Enabling" if enabled else "Disabling"))
self._enabled = enabled
# notify listeners that the painter has been enabled/disabled
self._notify_status_changed(enabled)
# paint or clear the database based on the change of status...
if enabled:
self._send_message(self.MSG_REPAINT)
else:
self._send_message(self.MSG_CLEAR)
#--------------------------------------------------------------------------
# Commands
#--------------------------------------------------------------------------
def terminate(self):
"""
Cleanup & terminate the painter.
"""
self._end_threads = True
self._msg_queue.put(self.MSG_TERMINATE)
try:
self._painting_worker.join()
except RuntimeError: # thread was never started...
pass
# best effort to free up resources & improve interpreter spindown
del self._painted_nodes
del self._painted_instructions
del self._status_changed_callbacks
def repaint(self):
"""
Paint coverage defined by the current database mappings.
"""
self._send_message(self.MSG_REPAINT)
def force_repaint(self):
"""
Force a coverage repaint of the current database mappings.
"""
self._send_message(self.MSG_FORCE_REPAINT)
def force_clear(self):
"""
Clear all paint from the current database (based on metadata)
"""
self._send_message(self.MSG_FORCE_CLEAR)
self.set_enabled(False)
def check_rebase(self):
"""
Perform a rebase on the painted data cache (if necessary).
"""
self._send_message(self.MSG_REBASE)
self._send_message(self.MSG_REPAINT)
def _send_message(self, message):
"""
Queue a painter command for execution.
"""
if not self._started:
return
self._msg_queue.put(message)
#--------------------------------------------------------------------------
# Commands
#--------------------------------------------------------------------------
def status_changed(self, callback):
"""
Subscribe a callback for coverage switch events.
"""
register_callback(self._status_changed_callbacks, callback)
def _notify_status_changed(self, status):
"""
Notify listeners of a coverage switch event.
"""
notify_callback(self._status_changed_callbacks, status)
#--------------------------------------------------------------------------
# Paint Primitives
#--------------------------------------------------------------------------
@abc.abstractmethod
def _paint_instructions(self, instructions):
"""
Paint instruction coverage defined by the current database mapping.
"""
pass
@abc.abstractmethod
def _clear_instructions(self, instructions):
"""
Clear paint from the given instructions.
"""
pass
@abc.abstractmethod
def _paint_nodes(self, nodes_coverage):
"""
Paint node coverage defined by the current database mappings.
"""
pass
@abc.abstractmethod
def _clear_nodes(self, nodes_metadata):
"""
Clear paint from the given graph nodes.
"""
pass
@abc.abstractmethod
def _refresh_ui(self):
"""
Refresh the disassembler UI to ensure paint is rendered.
"""
pass
@abc.abstractmethod
def _cancel_action(self, job):
"""
Cancel a paint action using something representing its job.
"""
pass
#------------------------------------------------------------------------------
# Painting - High Level
#------------------------------------------------------------------------------
def _priority_paint(self):
"""
Immediately repaint regions of the database visible to the user.
Return True upon completion, or False if interrupted.
"""
if self._streaming_instructions and self._streaming_nodes:
return True
# get current function / user location in the database
cursor_address = disassembler[self.lctx].get_current_address()
# attempt to paint the functions in the immediate cursor vicinity
result = self._priority_paint_functions(cursor_address)
# force a refresh *now* as this is a prority painting
self._refresh_ui()
# all done
return result
def _priority_paint_functions(self, target_address, neighbors=1):
"""
Paint functions in the immediate vicinity of the given address.
This will paint both the instructions & graph nodes of defined functions.
"""
db_metadata = self.director.metadata
db_coverage = self.director.coverage
blank_coverage = FunctionCoverage(BADADDR)
# get the function metadata for the function closest to our cursor
function_metadata = db_metadata.get_closest_function(target_address)
if not function_metadata:
return False
# select the range of functions around us that we would like to paint
func_num = db_metadata.get_function_index(function_metadata.address)
func_num_start = max(func_num - neighbors, 0)
func_num_end = min(func_num + neighbors + 1, len(db_metadata.functions) - 1)
# repaint the specified range of functions
for current_num in xrange(func_num_start, func_num_end):
# get the next function to paint
function_metadata = db_metadata.get_function_by_index(current_num)
if not function_metadata:
continue
# get the function coverage data for the target address
function_address = function_metadata.address
function_coverage = db_coverage.functions.get(function_address, blank_coverage)
if not self._streaming_nodes:
# clear nodes
must_clear = sorted(set(function_metadata.nodes) - set(function_coverage.nodes))
self._action_complete.clear()
self._clear_nodes(must_clear)
self._action_complete.wait()
# paint nodes
must_paint = sorted(function_coverage.nodes)
self._action_complete.clear()
self._paint_nodes(must_paint)
self._action_complete.wait()
if not self._streaming_instructions:
# clear instructions
must_clear = sorted(function_metadata.instructions - function_coverage.instructions)
self._action_complete.clear()
self._clear_instructions(must_clear)
self._action_complete.wait()
# paint instructions
must_paint = sorted(function_coverage.instructions)
self._action_complete.clear()
self._paint_instructions(must_paint)
self._action_complete.wait()
# paint finished successfully
return True
def _paint_database(self):
"""
Repaint the current database based on the current state.
"""
logger.debug("Painting database...")
# more code-friendly, readable aliases (db_XX == database_XX)
db_coverage = self.director.coverage
db_metadata = self.director.metadata
start = time.time()
#------------------------------------------------------------------
# initialize imagebase if it hasn't been already...
if self._imagebase == BADADDR:
self._imagebase = db_metadata.imagebase
# immediately paint user-visible regions of the database
if not self._priority_paint():
return False # a repaint was requested
#
# if the painter is not capable of 'streaming' the coverage paint,
# then we must explicitly paint the instructions & nodes here
#
if not self._streaming_instructions:
#
# TODO: 'partially painted nodes' might be a little funny / not
# working correctly in IDA if we ever disable instruction streaming...
#
# compute the painted instructions that will not get painted over
stale_instr = self._painted_instructions - db_coverage.coverage
stale_instr |= (self._painted_partial - db_coverage.partial_instructions)
# clear old instruction paint
if not self._async_action(self._clear_instructions, stale_instr):
return False # a repaint was requested
# paint new instructions
new_instr = sorted(db_coverage.coverage - self._painted_instructions)
if not self._async_action(self._paint_instructions, new_instr):
return False # a repaint was requested
if not self._streaming_nodes:
# compute the painted nodes that will not get painted over
stale_nodes = self._painted_nodes - viewkeys(db_coverage.nodes)
stale_nodes |= db_coverage.partial_nodes
# clear old node paint
if not self._async_action(self._clear_nodes, stale_nodes):
return False # a repaint was requested
# paint new nodes
new_nodes = sorted(viewkeys(db_coverage.nodes) - self._painted_nodes)
if not self._async_action(self._paint_nodes, new_nodes):
return False # a repaint was requested
#------------------------------------------------------------------
end = time.time()
logger.debug(" - Painting took %.2f seconds" % (end - start))
# paint finished successfully
return True
def _clear_database(self):
"""
Clear all paint from the current database using the known paint state.
"""
logger.debug("Clearing database paint...")
start = time.time()
#------------------------------------------------------------------
db_metadata = self.director.metadata
# clear all instructions
if not self._streaming_instructions:
if not self._async_action(self._clear_instructions, self._painted_instructions):
return False # a repaint was requested
# clear all nodes
if not self._streaming_nodes:
if not self._async_action(self._clear_nodes, self._painted_nodes):
return False # a repaint was requested
#------------------------------------------------------------------
end = time.time()
logger.debug(" - Database paint cleared in %.2f seconds..." % (end-start))
# sanity checks...
assert self._painted_nodes == set()
assert self._painted_instructions == set()
# paint finished successfully
return True
def _force_paint_database(self):
"""
Forcibly repaint the database.
"""
db_metadata = self.director.metadata
text = "Repainting the database..."
logger.debug(text)
is_modal = bool(disassembler.NAME != "IDA")
disassembler.execute_ui(disassembler.show_wait_box)(text, False)
start = time.time()
#------------------------------------------------------------------
# discard current / known paint state
self._painted_nodes = set()
self._painted_partial = set()
self._painted_instructions = set()
# paint the database...
self._paint_database()
#------------------------------------------------------------------
end = time.time()
logger.debug(" - Database repainted in %.2f seconds..." % (end-start))
time.sleep(.2) # XXX: this seems to fix a bug where the waitbox doesn't close if the paint is too fast??
disassembler.execute_ui(disassembler.hide_wait_box)()
# paint finished successfully
return True
def _force_clear_database(self):
"""
Forcibly clear the paint from all known database addresses.
"""
db_metadata = self.director.metadata
text = "Forcibly clearing all paint from database..."
logger.debug(text)
#
# NOTE: forcefully clearing the database of paint can take a long time
# in certain cases, so we want to block the user from doing anything
# to the database while we're working.
#
# we will pop up a waitbox to block them, but we have to be careful as
# a *modal* waitbox will conflict with IDA's processing of MFF_WRITE
# requests making it wait for the waitbox to close before processing
#
# therefore, we put in a little bodge wire here to make sure the
# waitbox is *not* modal for IDA... but will be in the normal case.
# it also helps that IDA will be busy processing our 'write' requests,
# so the UI will be mostly frozen to the user anyway!
#
is_modal = bool(disassembler.NAME != "IDA")
disassembler.execute_ui(disassembler.show_wait_box)(text, is_modal)
start = time.time()
#------------------------------------------------------------------
self._action_complete.clear()
self._clear_instructions(sorted(db_metadata.instructions))
self._action_complete.wait()
self._action_complete.clear()
self._clear_nodes(sorted(db_metadata.nodes))
self._action_complete.wait()
#------------------------------------------------------------------
end = time.time()
logger.debug(" - Database paint cleared in %.2f seconds..." % (end-start))
time.sleep(.2) # XXX: this seems to fix a bug where the waitbox doesn't close if the clear is too fast??
disassembler.execute_ui(disassembler.hide_wait_box)()
# paint finished successfully
return True
def _rebase_database(self):
"""
Rebase the active database paint.
TODO/XXX: there may be some edgecases where painting can be wrong if
a rebase occurs while the painter is running.
"""
db_metadata = self.director.metadata
# a rebase has not occurred
if not db_metadata.cached or (db_metadata.imagebase == self._imagebase):
return False
# compute the offset of the rebase
rebase_offset = db_metadata.imagebase - self._imagebase
# rebase the cached addresses of what we have painted
self._painted_nodes = set([address+rebase_offset for address in self._painted_nodes])
self._painted_instructions = set([address+rebase_offset for address in self._painted_instructions])
self._imagebase = db_metadata.imagebase
# a rebase has been observed
return True
#--------------------------------------------------------------------------
# Asynchronous Painting
#--------------------------------------------------------------------------
@catch_errors
def _async_database_painter(self):
"""
Asynchronous database painting worker loop.
"""
logger.debug("Starting DatabasePainter thread...")
#
# Asynchronous Database Painting Loop
#
while not self._end_threads:
# wait for the next command to come through
action = self._msg_queue.get()
# repaint the database based on the current state
if action == self.MSG_REPAINT:
result = self._paint_database()
# forcibly repaint the database based on the current state
elif action == self.MSG_FORCE_REPAINT:
result = self._force_paint_database()
# clear database base on the current state
elif action == self.MSG_CLEAR:
result = self._clear_database()
# clear all possible database paint
elif action == self.MSG_FORCE_CLEAR:
result = self._force_clear_database()
# check for a rebase of the painted data
elif action == self.MSG_REBASE:
result = self._rebase_database()
# thrown internally to escape a stale paint, just ignore
elif action == self.MSG_ABORT:
continue
# spin down the painting thread (this thread)
elif action == self.MSG_TERMINATE:
break
# unknown command
else:
logger.error("UNKNOWN COMMAND! %s" % str(action))
break
# refresh the UI to ensure paint changes are rendered
self._refresh_ui()
# thread exit
logger.debug("Exiting DatabasePainter thread...")
def _async_action(self, paint_action, work_iterable):
"""
Split a normal paint routine into interruptable chunks.
Internal routine for asynchrnous painting.
"""
CHUNK_SIZE = 1500 # somewhat arbitrary
# split the given nodes into multiple paints
for work_chunk in chunks(list(work_iterable), CHUNK_SIZE):
#
# reset the paint event signal so that it is ready for the next
# paint request. it will let us know when the asynchrnous paint
# action has completed in the IDA main thread
#
self._action_complete.clear()
#
# paint or unpaint a chunk of 'work' (nodes, or instructions) with
# the given paint function (eg, paint_nodes, clear_instructions)
#
paint_job = paint_action(work_chunk)
#
# wait for the asynchronous paint event to complete or a signal that
# we should end this thread (via end_threads)
#
while not (self._action_complete.wait(timeout=0.2) or self._end_threads):
continue
#
# our end_threads signal/bool can only originate from the main IDA
# thread (plugin termination). we make the assumption that no more
# MFF_WRITE requests (eg, 'paint_action') will get processed.
#
# we do a best effort to cancel the in-flight job (just in case)
# and return so we can exit the thread.
#
if self._end_threads:
self._cancel_action(paint_job)
return False
#
# the operation has been interrupted by a repaint request, bail
# immediately so that we can process the next repaint
#
if not self._msg_queue.empty():
return False
# operation completed successfully
return True
+1
View File
@@ -0,0 +1 @@
from .coverage_reader import CoverageReader
+114
View File
@@ -0,0 +1,114 @@
import abc
class CoverageFile(object):
"""
Templated class for Lighthouse-compatible code coverage file reader.
"""
__metaclass__ = abc.ABCMeta
@abc.abstractmethod
def __init__(self, filepath=None):
self.filepath = filepath
self.modules = {}
self._parse()
#--------------------------------------------------------------------------
# Parsing Routines
#--------------------------------------------------------------------------
@abc.abstractmethod
def _parse(self):
"""
Load and parse coverage data from the file defined by self.filepath
Within this function, a custom CoverageFile is expected to attempt to
parse the coverage file from disk. If the coverage file does not appear
to match the format expected by this parser -- that is okay.
Should this parser crash and burn, the CoverageReader will simply move
on to the next available parser and discard this attempt.
This function should *only* parse & categorize the coverage data that
it loads from disk. If this function returns without error, the
CoverageReader will attempt to call one of the get() functions later
to retrieve the data you have loaded.
The best coverage file formats will contain some sort of mapping
for the coverage data that ties it to a module or binary that was in
the instrumented process space.
If this mapping in known, then this function should strive to store
the coverage data in the self.modules dictionary, where
self.modules[module_name] = [ coverage_addresses ]
"""
raise NotImplementedError("Coverage parser not implemented")
#--------------------------------------------------------------------------
# Public
#--------------------------------------------------------------------------
#
# if you are writing a parser for a custom coverage file format, your
# parser is *REQUIRED* to implement one of the following routines.
#
# the CoverageReader well attempt to retrieve parsed data from this class
# using one of the function below.
#
def get_addresses(self, module_name=None):
"""
Return coverage data for the named module as absolute addresses.
If no name is given / available via self.modules, the trace is assumed
to be a an ABSOLUTE ADDRESS TRACE.
These are arugably the least flexible kind of traces available, but are
still provided as an option. This fuction should return a list of
integers representing absolute coverage addresses that match the open
disassembler database...
coverage_addresses = [address, address1, address2, ...]
"""
raise NotImplementedError("Absolute addresses not supported by this log format")
def get_offsets(self, module_name):
"""
Return coverage data for the named module as relative offets.
This function should return a list of integers representing the
relative offset of an executed instruction OR basic block from the
base of the requested module (module_name).
It is *okay* to return an instruction trace, OR a basic block trace
from thin function. Lighthoue will automatically detect basic block
based traces and 'explode' them into instruction traces.
coverage_data = [offset, offset2, offset3, ...]
"""
raise NotImplementedError("Relative addresses not supported by this log format")
def get_offset_blocks(self, module_name):
"""
Return coverage data for the named module in block form.
This function should return a list of tuples representing the coverage
for the requested module (module_name). The tuples must be in the form
of (offset, size).
offset - a relative offset from the module_name base address
size - the size of the instruction, block, or sequence executed
eg, if a basic block of 24 bytes in length at kernel32.dll+0x4182 was
executed, its tuple would be (0x4182, 24).
The complete list coverage data returned by thin function should be in
the following form:
coverage_data = [(offset, size), (offset1, size1), ...]
"""
raise NotImplementedError("Block form not supported by this log format")
@@ -0,0 +1,140 @@
import os
import sys
import inspect
import logging
import traceback
from .coverage_file import CoverageFile
from lighthouse.util.python import iteritems
from lighthouse.exceptions import CoverageParsingError
logger = logging.getLogger("Lighthouse.Reader")
MODULES_DIRECTORY = os.path.join(os.path.dirname(os.path.realpath(__file__)), "parsers")
class CoverageReader(object):
"""
Middleware to automatically parse and load different coverage file formats.
This class will dynamically load and make use of coverage file parsers
that subclass from the CoverageFile abstraction and live within the
reader's 'parsers' folder.
This should allow end-users to write parsers for custom coverage file
format without having to modify any of Lighthouse's existing code (ideally)
"""
def __init__(self):
self._installed_parsers = {}
self._import_parsers()
def open(self, filepath):
"""
Open and parse a coverage file from disk.
Returns a CoverageFile on success, or raises CoverageParsingError on failure.
"""
coverage_file = None
parse_failures = {}
# attempt to parse the given coverage file with each available parser
for name, parser in iteritems(self._installed_parsers):
logger.debug("Attempting parse with '%s'" % name)
# attempt to open/parse the coverage file with the given parser
try:
coverage_file = parser(filepath)
break
# log the exceptions for each parse failure
except Exception as e:
parse_failures[name] = traceback.format_exc()
logger.debug("| Parse FAILED - " + str(e))
#logger.exception("| Parse FAILED")
#
# if *all* the coverage file parsers failed, raise an exception with
# information for each failure (for debugging)
#
if not coverage_file:
raise CoverageParsingError(filepath, parse_failures)
# successful parse
logger.debug("| Parse OKAY")
return coverage_file
def _import_parsers(self):
"""
Scan and import coverage file parsers from the 'parsers' directory.
"""
target_subclass = CoverageFile
ignored_files = ["__init__.py"]
# loop through all the files in the parsers folder
for filename in os.listdir(MODULES_DIRECTORY):
# ignore specified files, and anything not *.py
if filename in ignored_files or filename.endswith(".py") == False:
continue
# attempt to load a CoverageFile format from the current *.py file
logger.debug("| Searching file %s" % filename)
parser_file = filename[:-3]
parser_class = self._locate_subclass(parser_file, target_subclass)
if not parser_class:
logger.warning("| - No object subclassing from %s found in %s..." \
% (target_subclass.__name__, parser_file))
continue
# instantiate and add the parser to our dict of imported parsers
logger.debug("| | Found %s" % parser_class.__name__)
self._installed_parsers[parser_class.__name__] = parser_class
logger.debug("+- Done dynamically importing parsers")
# return the number of modules successfully imported
return self._installed_parsers
def _locate_subclass(self, module_file, target_subclass):
"""
Return the first matching target_subclass in module_file.
This function is used to scan a specific file (module_file) in the
Lighthouse parsers directory for class definitions that subclass from
target_subclass.
We use this to dynmically import, locate, and return objects that are
utilizing our CoverageFile abstraction.
"""
module = None
module_class = None
# attempt to import the given filepath as a python module
try:
module = __import__("lighthouse.reader.parsers." + module_file, globals(), locals(), ['object'])
except Exception as e:
logger.exception("| - Parser import failed")
return None
#
# inspect the module for any classes that subclass from target_subclass
# eg: target_subclass == CoverageFile
#
class_members = inspect.getmembers(module, inspect.isclass)
for a_class in class_members:
# does the current class definition we're inspecting subclass
# from target_subclass? if so, it is a match
try:
if a_class[1].__bases__[0] == target_subclass:
module_class = a_class[1]
break
# this class does not subclass / etc / not interesting / ignore it
except IndexError as e:
pass
# return discovered parser or None
return module_class
@@ -1,100 +1,121 @@
#!/usr/bin/python
import os
import re
import sys
import mmap
import struct
import re
import collections
from ctypes import *
#
# I know people like to use this parser in their own projects, so this
# if/def makes it compatible with being imported or used outside Lighthouse
#
try:
from lighthouse.exceptions import CoverageMissingError
from lighthouse.reader.coverage_file import CoverageFile
g_lighthouse = True
except ImportError as e:
CoverageFile = object
g_lighthouse = False
#------------------------------------------------------------------------------
# drcov log parser
# DynamoRIO Drcov Log Parser
#------------------------------------------------------------------------------
class DrcovData(object):
class DrcovData(CoverageFile):
"""
A drcov log parser.
"""
def __init__(self, filepath=None):
# original filepath
def __init__(self, filepath=None):
self.filepath = filepath
# drcov header attributes
self.version = 0
self.flavor = None
self.flavor = None
# drcov module table
self.module_table_count = 0
self.module_table_count = 0
self.module_table_version = 0
self.modules = []
self.modules = {}
# drcov basic block data
self.bb_table_count = 0
self.bbs = []
self.bb_table_count = 0
self.bb_table_is_binary = True
self.basic_blocks = []
# parse the given filepath
self._parse_drcov_file(filepath)
# parse
if g_lighthouse:
super(DrcovData, self).__init__(filepath)
else:
self._parse()
#--------------------------------------------------------------------------
# Public
#--------------------------------------------------------------------------
def get_module(self, module_name, fuzzy=True):
def get_offsets(self, module_name):
"""
Get a module by its name.
Note that this is a 'fuzzy' lookup by default.
Return coverage data as basic block offsets for the named module.
"""
modules = self.modules.get(module_name, [])
if not modules:
return []
# fuzzy module name lookup
if fuzzy:
#
# I don't know if this should ever actually trigger, but if it does,
# it is a strange testcase to collect coverage against. It means that
# maybe the target library/module was loaded, unloaded, and reloaded?
#
# if someone ever actally triggers this, we can look into it :S
#
# attempt lookup using case-insensitive filename
for module in self.modules:
if module_name.lower() in module.filename.lower():
return module
if self.version > 2:
assert all(module.containing_id == modules[0].id for module in modules)
#
# no hits yet... let's cleave the extension from the given module
# name (if present) and try again
#
# extract the unique module ids that we need to collect blocks for
mod_ids = [module.id for module in modules]
if "." in module_name:
module_name = module_name.split(".")[0]
# loop through the coverage data and filter out data for the target ids
coverage_blocks = [bb.start for bb in self.bbs if bb.mod_id in mod_ids]
# attempt lookup using case-insensitive filename without extension
for module in self.modules:
if module_name.lower() in module.filename.lower():
return module
# return the filtered coverage blocks
return coverage_blocks
def get_offset_blocks(self, module_name):
"""
Return coverage data as basic blocks (offset, size) for the named module.
"""
modules = self.modules.get(module_name, [])
if not modules:
return []
# NOTE: see comment in get_offsets() for more info...
if self.version > 2:
assert all(module.containing_id == modules[0].id for module in modules)
# extract the unique module ids that we need to collect blocks for
mod_ids = [module.id for module in modules]
# loop through the coverage data and filter out data for the target ids
if self.version < 3:
coverage_blocks = [(bb.start, bb.size) for bb in self.bbs if bb.mod_id in mod_ids]
#
# drcov version 3 does not include the 'preferred' / sub-module base
# in the bb offset, so we must add that base offset before returning
# the block offsets to correctly normalize things
#
# it's unclear if the preferred_base for given sub-module segments
# will always be correct, so we opt to simply use the first segment
# in a given module as the base to compute the known runtime offset
#
# strict lookup
else:
for module in self.modules:
if module_name == module.filename:
return module
# no matching module exists
return None
def get_blocks_by_module(self, module_name):
"""
Extract coverage blocks pertaining to the named module.
"""
# locate the coverage that matches the given module_name
module = self.get_module(module_name)
# if we fail to find a module that matches the given name, bail
if not module:
raise ValueError("No coverage for module '%s' in log" % module_name)
# extract module id for speed
mod_id = module.id
# loop through the coverage data and filter out data for only this module
coverage_blocks = [(bb.start, bb.size) for bb in self.basic_blocks if bb.mod_id == mod_id]
mod_bases = dict([(module.id, module.start - modules[0].start) for module in modules])
coverage_blocks = [(mod_bases[bb.mod_id] + bb.start, bb.size) for bb in self.bbs if bb.mod_id in mod_ids]
# return the filtered coverage blocks
return coverage_blocks
@@ -103,21 +124,15 @@ class DrcovData(object):
# Parsing Routines - Top Level
#--------------------------------------------------------------------------
def _parse_drcov_file(self, filepath):
def _parse(self):
"""
Parse drcov coverage from the given log file.
"""
with open(filepath, "rb") as f:
with open(self.filepath, "rb") as f:
self._parse_drcov_header(f)
self._parse_module_table(f)
self._parse_bb_table(f)
def _parse_drcov_data(self, drcov_data):
"""
Parse drcov coverage from the given data blob.
"""
pass # TODO/DRCOV
#--------------------------------------------------------------------------
# Parsing Routines - Internals
#--------------------------------------------------------------------------
@@ -129,15 +144,15 @@ class DrcovData(object):
# parse drcov version from log
# eg: DRCOV VERSION: 2
version_line = f.readline().strip()
version_line = f.readline().decode('utf-8').strip()
self.version = int(version_line.split(":")[1])
# parse drcov flavor from log
# eg: DRCOV FLAVOR: drcov
flavor_line = f.readline().strip()
flavor_line = f.readline().decode('utf-8').strip()
self.flavor = flavor_line.split(":")[1]
assert self.version == 2, "Only drcov version 2 log files supported"
assert self.version == 2 or self.version == 3, "Only drcov versions 2 and 3 log files supported"
def _parse_module_table(self, f):
"""
@@ -163,7 +178,7 @@ class DrcovData(object):
# parse module table 'header'
# eg: Module Table: version 2, count 11
header_line = f.readline().strip()
header_line = f.readline().decode('utf-8').strip()
field_name, field_data = header_line.split(": ")
#assert field_name == "Module Table"
@@ -192,7 +207,7 @@ class DrcovData(object):
data_name, version = version_data.split(" ")
#assert data_name == "version"
self.module_table_version = int(version)
if not self.module_table_version in [2, 3, 4]:
if not self.module_table_version in [2, 3, 4, 5]:
raise ValueError("Unsupported (new?) drcov log format...")
# parse module count in table from 'count Y'
@@ -227,6 +242,12 @@ class DrcovData(object):
Mac/Linux:
'Columns: id, containing_id, start, end, entry, offset, path'
DynamoRIO v10.0.19734, table version 5:
Windows:
'Columns: id, containing_id, start, end, entry, offset, preferred_base, checksum, timestamp, path'
Mac/Linux:
'Columns: id, containing_id, start, end, entry, offset, preferred_base, path'
"""
# NOTE/COMPAT: there is no 'Columns' line for the v1 table...
@@ -235,7 +256,7 @@ class DrcovData(object):
# parse module table 'columns'
# eg: Columns: id, base, end, entry, checksum, timestamp, path
column_line = f.readline().strip()
column_line = f.readline().decode('utf-8').strip()
field_name, field_data = column_line.split(": ")
#assert field_name == "Columns"
@@ -248,11 +269,14 @@ class DrcovData(object):
"""
Parse drcov log modules in the module table from filestream.
"""
modules = collections.defaultdict(list)
# loop through each *expected* line in the module table and parse it
for i in xrange(self.module_table_count):
module = DrcovModule(f.readline().strip(), self.module_table_version)
self.modules.append(module)
for i in range(self.module_table_count):
module = DrcovModule(f.readline().decode('utf-8').strip(), self.module_table_version)
modules[module.filename].append(module)
self.modules = modules
def _parse_bb_table(self, f):
"""
@@ -268,7 +292,7 @@ class DrcovData(object):
# parse basic block table 'header'
# eg: BB Table: 2792 bbs
header_line = f.readline().strip()
header_line = f.readline().decode('utf-8').strip()
field_name, field_data = header_line.split(": ")
#assert field_name == "BB Table"
@@ -279,7 +303,7 @@ class DrcovData(object):
# peek at the next few bytes to determine if this is a binary bb table.
# An ascii bb table will have the line: 'module id, start, size:'
token = "module id"
token = b"module id"
saved_position = f.tell()
# is this an ascii table?
@@ -297,30 +321,40 @@ class DrcovData(object):
"""
Parse drcov log basic block table entries from filestream.
"""
# allocate the ctypes structure array of basic blocks
self.basic_blocks = (DrcovBasicBlock * self.bb_table_count)()
self.bbs = (DrcovBasicBlock * self.bb_table_count)()
# read binary basic block entries directly into the newly allocated array
if self.bb_table_is_binary:
# read the basic block entries directly into the newly allocated array
f.readinto(self.basic_blocks)
f.readinto(self.bbs)
else: # let's parse the text records
text_entry = f.readline().strip()
# parse the plaintext basic block entries one by one
else:
self._parse_bb_table_text_entries(f)
if text_entry != "module id, start, size:":
raise ValueError("Invalid BB header: %r" % text_entry)
def _parse_bb_table_text_entries(self, f):
"""
Parse drcov log basic block table text entries from filestream.
"""
table_header = f.readline().decode('utf-8').strip()
pattern = re.compile(r"^module\[\s*(?P<mod>[0-9]+)\]\:\s*(?P<start>0x[0-9a-f]+)\,\s*(?P<size>[0-9]+)$")
for basic_block in self.basic_blocks:
text_entry = f.readline().strip()
if table_header != "module id, start, size:":
raise ValueError("Invalid BB header: %r" % table_header)
match = pattern.match(text_entry)
if not match:
raise ValueError("Invalid BB entry: %r" % text_entry)
pattern = re.compile(r"^module\[\s*(?P<mod>[0-9]+)\]\:\s*(?P<start>0x[0-9a-fA-F]+)\,\s*(?P<size>[0-9]+)$")
for i, bb in enumerate(self.bbs):
text_entry = f.readline().decode('utf-8').strip()
if not text_entry:
continue
basic_block.start = int(match.group("start"), 16)
basic_block.size = int(match.group("size"), 10)
basic_block.mod_id = int(match.group("mod"), 10)
match = pattern.match(text_entry)
if not match:
raise ValueError("Invalid BB entry: %r" % text_entry)
bb.start = int(match.group("start"), 16)
bb.size = int(match.group("size"), 10)
bb.mod_id = int(match.group("mod"), 10)
#------------------------------------------------------------------------------
# drcov module parser
@@ -371,6 +405,8 @@ class DrcovModule(object):
self._parse_module_v3(data)
elif version == 4:
self._parse_module_v4(data)
elif version == 5:
self._parse_module_v5(data)
else:
raise ValueError("Unknown module format (v%u)" % version)
@@ -381,7 +417,7 @@ class DrcovModule(object):
self.id = int(data[0])
self.size = int(data[1])
self.path = str(data[2])
self.filename = os.path.basename(self.path)
self.filename = os.path.basename(self.path.replace('\\', os.sep))
def _parse_module_v2(self, data):
"""
@@ -396,7 +432,7 @@ class DrcovModule(object):
self.timestamp = int(data[5], 16)
self.path = str(data[-1])
self.size = self.end-self.base
self.filename = os.path.basename(self.path)
self.filename = os.path.basename(self.path.replace('\\', os.sep))
def _parse_module_v3(self, data):
"""
@@ -407,12 +443,12 @@ class DrcovModule(object):
self.base = int(data[2], 16)
self.end = int(data[3], 16)
self.entry = int(data[4], 16)
if len(data) == 7: # Windows Only
if len(data) > 7: # Windows Only
self.checksum = int(data[5], 16)
self.timestamp = int(data[6], 16)
self.path = str(data[-1])
self.size = self.end-self.base
self.filename = os.path.basename(self.path)
self.filename = os.path.basename(self.path.replace('\\', os.sep))
def _parse_module_v4(self, data):
"""
@@ -424,12 +460,31 @@ class DrcovModule(object):
self.end = int(data[3], 16)
self.entry = int(data[4], 16)
self.offset = int(data[5], 16)
if len(data) == 7: # Windows Only
if len(data) > 8: # Windows Only
self.checksum = int(data[6], 16)
self.timestamp = int(data[7], 16)
self.path = str(data[-1])
self.size = self.end-self.base
self.filename = os.path.basename(self.path)
self.filename = os.path.basename(self.path.replace('\\', os.sep))
def _parse_module_v5(self, data):
"""
Parse a module table v5 entry.
"""
self.id = int(data[0])
self.containing_id = int(data[1])
self.base = int(data[2], 16)
self.end = int(data[3], 16)
self.entry = int(data[4], 16)
self.offset = int(data[5], 16)
self.preferred_base = int(data[6], 16)
if len(data) > 8: # Windows Only
self.checksum = int(data[7], 16)
self.timestamp = int(data[8], 16)
self.path = str(data[-1])
self.size = self.end-self.base
self.filename = os.path.basename(self.path.replace('\\', os.sep))
#------------------------------------------------------------------------------
# drcov basic block parser
@@ -468,10 +523,11 @@ if __name__ == "__main__":
# base usage
if argc < 2:
print "usage: %s <coverage filename>" % os.path.basename(sys.argv[0])
print("usage: {} <coverage filename>".format(os.path.basename(sys.argv[0])))
sys.exit()
# attempt file parse
x = DrcovData(argv[1])
for bb in x.basic_blocks:
print "0x%08x" % bb.start
for bb in x.bbs:
print("0x{:08x}".format(bb.start))
@@ -0,0 +1,42 @@
import os
import collections
from ..coverage_file import CoverageFile
class ModOffData(CoverageFile):
"""
A module+offset log parser.
"""
def __init__(self, filepath):
super(ModOffData, self).__init__(filepath)
#--------------------------------------------------------------------------
# Public
#--------------------------------------------------------------------------
def get_offsets(self, module_name):
return self.modules.get(module_name, {}).keys()
#--------------------------------------------------------------------------
# Parsing Routines - Top Level
#--------------------------------------------------------------------------
def _parse(self):
"""
Parse modoff coverage from the given log file.
"""
modules = collections.defaultdict(lambda: collections.defaultdict(int))
with open(self.filepath) as f:
for line in f:
trimmed = line.strip()
# skip empty lines
if not len(trimmed): continue
# comments can start with ';' or '#'
if trimmed[0] in [';', '#']: continue
module_name, bb_offset = line.rsplit("+", 1)
modules[module_name][int(bb_offset, 16)] += 1
self.modules = modules
@@ -0,0 +1,82 @@
import collections
from ..coverage_file import CoverageFile
# 'known' instruction pointer labels from Tenet traces
INSTRUCTION_POINTERS = ['EIP', 'RIP', 'PC']
class TenetData(CoverageFile):
"""
A Tenet trace log parser.
"""
def __init__(self, filepath):
self._hitmap = {}
super(TenetData, self).__init__(filepath)
#--------------------------------------------------------------------------
# Public
#--------------------------------------------------------------------------
def get_addresses(self, module_name=None):
return self._hitmap.keys()
#--------------------------------------------------------------------------
# Parsing Routines - Top Level
#--------------------------------------------------------------------------
def _parse(self):
"""
Parse absolute instruction addresses from the given Tenet trace.
"""
hitmap = collections.defaultdict(int)
with open(self.filepath) as f:
while True:
# read 128mb chunks of 'lines' from the file
lines = f.readlines(1024 * 1024 * 128)
# no more lines to process, break
if not lines:
break
# parse the instruction addresses from lines, into the hitmap
self._process_lines(lines, hitmap)
# save the hitmap if we completed parsing without crashing
self._hitmap = hitmap
def _process_lines(self, lines, hitmap):
"""
Parse instruction addresses out of the given text lines.
"""
for line in lines:
# split the line (an execution delta) into its individual entries
delta = line.split(",")
# process each item (a name=value pair) in the execution delta
for item in delta:
# split name/value pair, and normalize the name for matching
name, value = item.split("=")
name = name.upper()
# ignore entries that are not the instruction pointer
if not name in INSTRUCTION_POINTERS:
continue
# save the parsed instruction pointer address to the hitmap
address = int(value, 16)
hitmap[address] += 1
# break beacuse we don't expect two IP's on the same line
break
# continue to the next line
# ...
# done parsing this chunk of lines
return
@@ -0,0 +1,32 @@
import collections
from ..coverage_file import CoverageFile
class TraceData(CoverageFile):
"""
An instruction (or basic block) address trace log parser.
"""
def __init__(self, filepath):
self._hitmap = {}
super(TraceData, self).__init__(filepath)
#--------------------------------------------------------------------------
# Public
#--------------------------------------------------------------------------
def get_addresses(self, module_name=None):
return self._hitmap.keys()
#--------------------------------------------------------------------------
# Parsing Routines - Top Level
#--------------------------------------------------------------------------
def _parse(self):
"""
Parse absolute address coverage from the given log file.
"""
hitmap = collections.defaultdict(int)
with open(self.filepath) as f:
for line in f:
hitmap[int(line, 16)] += 1
self._hitmap = hitmap
+4
View File
@@ -0,0 +1,4 @@
from .palette import LighthousePalette
from .coverage_xref import CoverageXref
from .module_selector import ModuleSelector
from .coverage_overview import CoverageOverview
@@ -1,4 +1,6 @@
import logging
import weakref
from lighthouse.util import *
from lighthouse.util.qt import *
from lighthouse.util.disassembler import disassembler
@@ -41,40 +43,38 @@ class CoverageComboBox(QtWidgets.QComboBox):
# configure the widget for use
self._ui_init()
self.refresh_theme()
#--------------------------------------------------------------------------
# QComboBox Overloads
#--------------------------------------------------------------------------
def mousePressEvent(self, e):
def mouseReleaseEvent(self, e):
"""
Capture mouse click events to the QComboBox.
Capture mouse release events on the QComboBox.
"""
# get the widget currently beneath the given mouse event
# get the widget currently beneath the mouse event being handled
hovering = self.childAt(e.pos())
#
# if the hovered widget is the 'head' of the QComboBox, we want to
# move any mouse clicks to appear like it was on the dropdown arrow
# box on the right side.
# if the hovered widget is the 'head' of the QComboBox, we assume
# the user has clicked it and should show the dropwdown 'popup'
#
# this is to satisfy some internal QComboBox Qt logic, allowing us
# to collapse and expand an 'editable' QComboBox by clicking anywhere
# on the 'head' (the read-only QLineEdit)
# we must showPopup() ourselves because internal Qt logic for
# 'editable' comboboxes try to enter an editing mode for the field
# rather than expanding the dropdown.
#
# this is basically dirty-hax
# if you don't remember, our combobox is marked 'editable' to satisfy
# some internal Qt logic so that our 'Windows' draw style is used
#
if hovering == self.lineEdit():
new_pos = QtCore.QPoint(
self.rect().right() - 10,
self.rect().height()/2
)
logger.debug("Moved click %s --> %s" % (e.pos(), new_pos))
e = move_mouse_event(e, new_pos)
self.showPopup()
e.accept()
return
# handle the event (possibly moved) as it normally would be
# handle any other events as they normally should be
super(CoverageComboBox, self).mousePressEvent(e)
#--------------------------------------------------------------------------
@@ -88,7 +88,7 @@ class CoverageComboBox(QtWidgets.QComboBox):
# initialize a monospace font to use with our widget(s)
self._font = MonospaceFont()
self._font.setPointSizeF(normalize_to_dpi(9))
self._font.setPointSizeF(normalize_to_dpi(10))
self._font_metrics = QtGui.QFontMetricsF(self._font)
self.setFont(self._font)
@@ -109,7 +109,6 @@ class CoverageComboBox(QtWidgets.QComboBox):
self.lineEdit().setFont(self._font)
self.lineEdit().setReadOnly(True) # text can't be edited
self.lineEdit().setEnabled(False) # text can't be selected
self.setMaximumHeight(self._font_metrics.height()*1.75)
#
# the combobox will pick a size based on its contents when it is first
@@ -119,18 +118,7 @@ class CoverageComboBox(QtWidgets.QComboBox):
self.setSizeAdjustPolicy(QtWidgets.QComboBox.AdjustToContentsOnFirstShow)
self.setSizePolicy(QtWidgets.QSizePolicy.Ignored, QtWidgets.QSizePolicy.Ignored)
#
# the purpose of this stylesheet is to pad the visible selection text
# in the combobox 'head' on first show. The reason being is that
# without this, the text for the selected coverage will lapse behind
# the combobox dropdown arrow (which is Qt by design???)
#
# I don't like the tail of the text disappearing behind this silly
# dropdown arrow, therefore we pad the right side of the combobox.
#
self.setStyleSheet("QComboBox { padding: 0 2ex 0 2ex; }")
self.setMaximumHeight(int(self._font_metrics.height()*1.75))
# draw the QComboBox with a 'Windows'-esque style
self.setStyle(QtWidgets.QStyleFactory.create("Windows"))
@@ -204,18 +192,10 @@ class CoverageComboBox(QtWidgets.QComboBox):
# event, (it looks weird) so clear the table/dropdown highlights now
#
# NOTE/COMPAT
if USING_PYQT5:
self.view().selectionModel().setCurrentIndex(
QtCore.QModelIndex(),
QtCore.QItemSelectionModel.ClearAndSelect
)
else:
self.view().selectionModel().setCurrentIndex(
QtCore.QModelIndex(),
QtGui.QItemSelectionModel.ClearAndSelect
)
self.view().selectionModel().setCurrentIndex(
QtCore.QModelIndex(),
QtCore.QItemSelectionModel.ClearAndSelect
)
#
# the deletion of an entry will shift all the entries beneath it up
@@ -249,6 +229,36 @@ class CoverageComboBox(QtWidgets.QComboBox):
"""
self._internal_refresh()
@disassembler.execute_ui
def refresh_theme(self):
"""
Refresh UI facing elements to reflect the current theme.
"""
palette = self._director.palette
self.view().refresh_theme()
# configure the combobox's top row / visible dropdown
self.lineEdit().setStyleSheet(
"QLineEdit { "
" border: none;"
" padding: 0 0 0 2ex;"
" margin: 0;"
" background-color: %s;" % palette.combobox_background.name() +
"}"
)
# style the combobox dropdown
self.setStyleSheet(
"QComboBox {"
" color: %s;" % palette.combobox_text.name() +
" border: 1px solid %s;" % palette.combobox_border.name() +
" padding: 0;"
"} "
"QComboBox:hover, QComboBox:focus {"
" border: 1px solid %s;" % palette.combobox_border_focus.name() +
"}"
)
@disassembler.execute_ui
def _internal_refresh(self):
"""
@@ -289,17 +299,91 @@ class CoverageComboBoxView(QtWidgets.QTableView):
def __init__(self, model, parent=None):
super(CoverageComboBoxView, self).__init__(parent)
self.setObjectName(self.__class__.__name__)
self._combobox = weakref.proxy(parent)
self._timer = None
# install the given data model into the table view
self.setModel(model)
# initialize UI elements
self._ui_init()
self.refresh_theme()
#--------------------------------------------------------------------------
# QTableView Overloads
#--------------------------------------------------------------------------
def showEvent(self, e):
"""
Show the QComboBox dropdown/popup.
"""
#
# the next line of code will prevent the combobox 'head' from getting
# any mouse actions now that the popup/dropdown is visible.
#
# this is pretty aggressive, but it will allow the user to 'collapse'
# the combobox dropdown while it is in an expanded state by simply
# clicking the combobox head as one can do to expand it.
#
# the reason this dirty trick is able to simulate a 'collapsing click'
# is because the user clicks 'outside' the popup/dropdown which
# automatically closes it. if the click was on the combobox head, it
# is simply ignored because we set this attribute!
#
# when the popup is closing, we undo this action in hideEvent().
#
# we have to use this workaround because we are using an 'editable' Qt
# combobox which behaves differently to clicks than a normal combobox.
#
# NOTE: we have to do this here in the tableview because the combobox's
# showPopup() and hidePopup() do not always trigger symmetrically.
#
# for example, hidePopup() was not being triggered when focus was lost
# via virutal desktop switch, and other external focus changes. this
# is really bad, because the combobox would get stuck *closed* as it
# was never re-enabled for mouse events
#
self._combobox.setAttribute(QtCore.Qt.WA_TransparentForMouseEvents)
def hideEvent(self, e):
"""
Hide the QComboBox dropdown/popup.
"""
#
# the combobox popup is now hidden / collapsed. the combobox head needs
# to be re-enlightened to direct mouse clicks (eg, to expand it). this
# undos the setAttribute action in showPopup() above.
#
# if the coverage combobox is *not* visible, the coverage window is
# probably being closed / deleted. but just in case, we should attempt
# to restore the combobox's ability to accept clicks before bailing.
#
# this fixes a bug / Qt warning first printed in IDA 7.4 where 'self'
# (the comobobox) would be deleted by the time the 100ms timer in the
# 'normal' case fires below
#
if not self._combobox.isVisible():
self._combobox.setAttribute(QtCore.Qt.WA_TransparentForMouseEvents, False)
return
#
# in the more normal case, the comobobox is simply being collapsed
# by the user clicking it, or clicking away from it.
#
# we use a short timer of 100ms to ensure the 'hiding' of the dropdown
# and its associated click are processed first. aftwards, it is safe to
# begin accepting clicks again.
#
self._timer = QtCore.QTimer.singleShot(100, self.__hidePopup_setattr)
def __hidePopup_setattr(self):
self._combobox.setAttribute(QtCore.Qt.WA_TransparentForMouseEvents, False)
def leaveEvent(self, e):
"""
Overload the mouse leave event.
@@ -326,27 +410,13 @@ class CoverageComboBoxView(QtWidgets.QTableView):
"""
Initialize UI elements.
"""
palette = self.model()._director._palette
# initialize a monospace font to use with our widget(s)
self._font = MonospaceFont()
self._font.setPointSizeF(normalize_to_dpi(9))
self._font.setPointSizeF(normalize_to_dpi(10))
self._font_metrics = QtGui.QFontMetricsF(self._font)
self.setFont(self._font)
# widget style
self.setStyleSheet(
"QTableView {"
" background-color: %s;" % palette.combobox_bg.name() +
" color: %s;" % palette.combobox_fg.name() +
" selection-background-color: %s;" % palette.combobox_selection_bg.name() +
" selection-color: %s;" % palette.combobox_selection_fg.name() +
" margin: 0; outline: none;"
"} "
"QTableView::item{ padding: 0.5ex; } "
"QTableView::item:focus { padding: 0; }"
)
# hide dropdown table headers, and default grid
self.horizontalHeader().setVisible(False)
self.verticalHeader().setVisible(False)
@@ -362,20 +432,12 @@ class CoverageComboBoxView(QtWidgets.QTableView):
hh = self.horizontalHeader()
#
# NOTE/COMPAT:
# - set the coverage name column to be stretchy and as tall as the text
# - make the 'X' icon column fixed width
#
if USING_PYQT5:
hh.setSectionResizeMode(0, QtWidgets.QHeaderView.Stretch)
hh.setSectionResizeMode(1, QtWidgets.QHeaderView.Fixed)
vh.setSectionResizeMode(QtWidgets.QHeaderView.ResizeToContents)
else:
hh.setResizeMode(0, QtWidgets.QHeaderView.Stretch)
hh.setResizeMode(1, QtWidgets.QHeaderView.Fixed)
vh.setResizeMode(QtWidgets.QHeaderView.ResizeToContents)
hh.setSectionResizeMode(0, QtWidgets.QHeaderView.Stretch)
vh.setSectionResizeMode(QtWidgets.QHeaderView.ResizeToContents)
vh.setMinimumSectionSize(0)
# get the column width hint from the model for the 'X' delete column
@@ -386,10 +448,12 @@ class CoverageComboBoxView(QtWidgets.QTableView):
)
# set the 'X' delete icon column width to a fixed size based on the hint
hh.setMinimumSectionSize(icon_column_width)
hh.resizeSection(COLUMN_DELETE, icon_column_width)
hh.setSectionResizeMode(1, QtWidgets.QHeaderView.ResizeToContents)
# install a delegate to do some custom painting against the combobox
self.setItemDelegate(ComboBoxDelegate())
self.setItemDelegate(ComboBoxDelegate(self))
#--------------------------------------------------------------------------
# Refresh
@@ -419,6 +483,28 @@ class CoverageComboBoxView(QtWidgets.QTableView):
else:
self.setSpan(row, 0, 0, model.columnCount())
@disassembler.execute_ui
def refresh_theme(self):
"""
Refresh UI facing elements to reflect the current theme.
"""
palette = self.model()._director.palette
self.setStyleSheet(
"QTableView {"
" background-color: %s;" % palette.combobox_background.name() +
" color: %s;" % palette.combobox_text.name() +
" margin: 0; outline: none;"
" border: 1px solid %s; " % palette.shell_border.name() +
"} "
"QTableView::item { " +
" padding: 0.5ex; border: 0; "
"} "
"QTableView::item:focus { " +
" background-color: %s; " % palette.combobox_selection_background.name() +
" color: %s; " % palette.combobox_selection_text.name() +
"} "
)
#------------------------------------------------------------------------------
# Coverage ComboBox - TableModel
#------------------------------------------------------------------------------
@@ -439,14 +525,14 @@ class CoverageComboBoxModel(QtCore.QAbstractTableModel):
# initialize a monospace font to use with our widget(s)
self._font = MonospaceFont()
self._font.setPointSizeF(normalize_to_dpi(9))
self._font.setPointSizeF(normalize_to_dpi(10))
self._font_metrics = QtGui.QFontMetricsF(self._font)
# load the raw 'X' delete icon from disk
delete_icon = QtGui.QPixmap(plugin_resource("icons/delete_coverage.png"))
# compute the appropriate size for the deletion icon
icon_height = self._font_metrics.height()*0.75
icon_height = int(self._font_metrics.height()*0.75)
icon_width = icon_height
# scale the icon as appropriate (very likely scaling it down)
@@ -544,8 +630,13 @@ class CoverageComboBoxModel(QtCore.QAbstractTableModel):
elif role == QtCore.Qt.TextAlignmentRole:
return QtCore.Qt.AlignVCenter | QtCore.Qt.AlignLeft
# combobox header, padded with " " to account for dropdown arrow overlap
elif role == QtCore.Qt.EditRole:
if index.column() == COLUMN_COVERAGE_STRING and index.row() != self._seperator_index:
return self._director.get_coverage_string(self._entries[index.row()]) + " "
# data display request
elif role in [QtCore.Qt.DisplayRole, QtCore.Qt.EditRole]:
elif role == QtCore.Qt.DisplayRole:
if index.column() == COLUMN_COVERAGE_STRING and index.row() != self._seperator_index:
return self._director.get_coverage_string(self._entries[index.row()])
@@ -668,12 +759,11 @@ class ComboBoxDelegate(QtWidgets.QStyledItemDelegate):
dropdown table in the Coverage ComboBox a bit more to our liking.
"""
def __init__(self, parent=None):
def __init__(self, parent):
super(ComboBoxDelegate, self).__init__(parent)
# painting property definitions
self._grid_color = QtGui.QColor(0x909090)
self._separator_color = QtGui.QColor(0x505050)
self._grid_color = parent.model()._director.palette.shell_border
def sizeHint(self, option, index):
"""
@@ -692,13 +782,15 @@ class ComboBoxDelegate(QtWidgets.QStyledItemDelegate):
if index.data(QtCore.Qt.AccessibleDescriptionRole) == ENTRY_USER:
painter.save()
painter.setPen(self._grid_color)
final_entry = (index.sibling(index.row()+1, 0).row() == -1)
# draw the grid line beneath the current row (a coverage entry)
tweak = QtCore.QPoint(0, 1) # 1px tweak provides better spacing
painter.drawLine(
option.rect.bottomLeft() + tweak,
option.rect.bottomRight() + tweak
)
if not final_entry:
painter.drawLine(
option.rect.bottomLeft() + tweak,
option.rect.bottomRight() + tweak
)
#
# now we will re-draw the grid line *above* the current entry,
@@ -714,18 +806,6 @@ class ComboBoxDelegate(QtWidgets.QStyledItemDelegate):
painter.restore()
# custom paint the separator entry between special & normal coverage
if index.data(QtCore.Qt.AccessibleDescriptionRole) == SEPARATOR:
painter.save()
painter.setPen(self._separator_color)
painter.drawRect(
option.rect
)
painter.restore()
# nothing else to paint for the separator entry
return
# custom paint the 'X' icon where applicable
if index.data(QtCore.Qt.DecorationRole):
@@ -747,9 +827,20 @@ class ComboBoxDelegate(QtWidgets.QStyledItemDelegate):
# draw the icon to the column
painter.drawPixmap(destination_rect, pixmap)
return
# nothing else to paint for the icon column entry
# custom paint the separator entry between special & normal coverage
if index.data(QtCore.Qt.AccessibleDescriptionRole) == SEPARATOR:
painter.save()
painter.setPen(self._grid_color)
painter.drawRect(
option.rect
)
painter.restore()
# nothing else to paint for the separator entry
return
# pass through to the standard painting
super(ComboBoxDelegate, self).paint(painter, option, index)
@@ -4,7 +4,8 @@ import weakref
from lighthouse.util.qt import *
from lighthouse.util.misc import plugin_resource
from lighthouse.util.disassembler import disassembler, DockableWindow
from lighthouse.util.disassembler import disassembler
from lighthouse.composer import ComposingShell
from lighthouse.ui.coverage_table import CoverageTableView, CoverageTableModel, CoverageTableController
from lighthouse.ui.coverage_combobox import CoverageComboBox
@@ -16,22 +17,24 @@ logger = logging.getLogger("Lighthouse.UI.Overview")
# Coverage Overview
#------------------------------------------------------------------------------
class CoverageOverview(DockableWindow):
class CoverageOverview(object):
"""
The Coverage Overview Widget.
"""
def __init__(self, core):
super(CoverageOverview, self).__init__(
"Coverage Overview",
plugin_resource(os.path.join("icons", "overview.png"))
)
self._core = core
self._visible = False
def __init__(self, lctx, widget):
self.lctx = lctx
self.widget = widget
self.director = self.lctx.director
self.lctx.coverage_overview = self
self.initialized = False
self._refreshed_callback = None
# see the EventProxy class below for more details
self._events = EventProxy(self)
self._widget.installEventFilter(self._events)
self.widget.installEventFilter(self._events)
# plugin_resource(os.path.join("icons", "overview.png"))
# initialize the plugin UI
self._ui_init()
@@ -39,32 +42,51 @@ class CoverageOverview(DockableWindow):
# refresh the data UI such that it reflects the most recent data
self.refresh()
# register for cues from the director
self._refreshed_callback = self.director.refreshed(self.refresh)
#--------------------------------------------------------------------------
# Pseudo Widget Functions
#--------------------------------------------------------------------------
def show(self):
"""
Show the CoverageOverview UI / widget.
"""
self.refresh()
super(CoverageOverview, self).show()
self._visible = True
@property
def name(self):
if not self.widget:
return "Coverage Overview"
return self.widget.name
@property
def visible(self):
if not self.widget:
return False
return self.widget.visible
def terminate(self):
"""
The CoverageOverview is being hidden / deleted.
"""
self._visible = False
if self.widget is None:
return
if self._refreshed_callback:
self.director.unregister_refreshed(self._refreshed_callback)
self._refreshed_callback = None
if self.lctx.coverage_overview is self:
self.lctx.coverage_overview = None
self.initialized = False
self._combobox = None
self._shell = None
self._toolbar = None
self._table_view = None
self._table_controller = None
self._table_model = None
self._widget = None
def isVisible(self):
return self._visible
self._settings_button = None
self._settings_menu = None
self._shell_elements = None
self._events = None
self.widget = None
#--------------------------------------------------------------------------
# Initialization - UI
@@ -87,13 +109,9 @@ class CoverageOverview(DockableWindow):
"""
Initialize the coverage table.
"""
self._table_model = CoverageTableModel(self._core.director, self._widget)
self._table_controller = CoverageTableController(self._table_model)
self._table_view = CoverageTableView(
self._table_controller,
self._table_model,
self._widget
)
self._table_model = CoverageTableModel(self.lctx, self.widget)
self._table_controller = CoverageTableController(self.lctx, self._table_model)
self._table_view = CoverageTableView(self._table_controller, self._table_model, self.widget)
def _ui_init_toolbar(self):
"""
@@ -124,18 +142,23 @@ class CoverageOverview(DockableWindow):
# the composing shell
self._shell = ComposingShell(
self._core.director,
self.lctx,
weakref.proxy(self._table_model),
weakref.proxy(self._table_view)
)
# the coverage combobox
self._combobox = CoverageComboBox(self._core.director)
self._combobox = CoverageComboBox(self.director)
# the splitter to make the shell / combobox resizable
self._shell_elements = QtWidgets.QSplitter(QtCore.Qt.Horizontal)
self._shell_elements.setStyleSheet(
"""
QSplitter
{
border: none;
}
QSplitter::handle
{
background-color: #909090;
@@ -172,13 +195,13 @@ class CoverageOverview(DockableWindow):
self._settings_button.setStyleSheet("QToolButton::menu-indicator{image: none;}")
# settings menu
self._settings_menu = TableSettingsMenu(self._widget)
self._settings_menu = TableSettingsMenu(self.widget)
def _ui_init_signals(self):
"""
Connect UI signals.
"""
self._settings_menu.connect_signals(self._table_controller, self._core)
self._settings_menu.connect_signals(self._table_controller, self.lctx)
self._settings_button.clicked.connect(self._ui_show_settings)
def _ui_layout(self):
@@ -188,12 +211,12 @@ class CoverageOverview(DockableWindow):
# layout the major elements of our widget
layout = QtWidgets.QGridLayout()
layout.setSpacing(get_dpi_scale())
layout.setSpacing(int(get_dpi_scale()*5))
layout.addWidget(self._table_view)
layout.addWidget(self._toolbar)
# apply the layout to the containing form
self._widget.setLayout(layout)
self.widget.setLayout(layout)
#--------------------------------------------------------------------------
# Signal Handlers
@@ -208,8 +231,8 @@ class CoverageOverview(DockableWindow):
-1*self._settings_menu.sizeHint().height()
)
center = QtCore.QPoint(
self._settings_button.sizeHint().width()/2,
self._settings_button.sizeHint().height()/2
int(self._settings_button.sizeHint().width()/2),
int(self._settings_button.sizeHint().height()/2)
)
where = self._settings_button.mapToGlobal(center+delta)
self._settings_menu.popup(where)
@@ -223,10 +246,26 @@ class CoverageOverview(DockableWindow):
"""
Refresh the Coverage Overview.
"""
if not (self._table_model and self._shell and self._combobox):
return
self._table_model.refresh()
self._shell.refresh()
self._combobox.refresh()
@disassembler.execute_ui
def refresh_theme(self):
"""
Update visual elements based on theme change.
"""
if not (self._table_view and self._table_model and self._shell and self._combobox):
return
self._table_view.refresh_theme()
self._table_model.refresh_theme()
self._shell.refresh_theme()
self._combobox.refresh_theme()
#------------------------------------------------------------------------------
# Qt Event Filter
#------------------------------------------------------------------------------
@@ -235,9 +274,20 @@ debugger_docked = False
class EventProxy(QtCore.QObject):
#
# NOTE/COMPAT: QtCore.QEvent.Destroy not in IDA7? Just gonna ship our own...
# - https://doc.qt.io/qt-5/qevent.html#Type-enum
#
EventShow = 17
EventDestroy = 16
EventLayoutRequest = 76
EventUpdateLater = 78
def __init__(self, target):
super(EventProxy, self).__init__()
self._target = target
self._target = weakref.proxy(target)
self._first_hit = True
def eventFilter(self, source, event):
@@ -246,19 +296,74 @@ class EventProxy(QtCore.QObject):
# cleanup after ourselves in the interest of stability
#
if int(event.type()) == 16: # NOTE/COMPAT: QtCore.QEvent.Destroy not in IDA7?
if int(event.type()) == self.EventDestroy:
source.removeEventFilter(self)
#
# XXX/V35: This is pretty hacky annoying stuff, but the lifetime
# of the CoverageOverview widget is managed internally by binja
# and gets deleted/cleaned up *after* a database is closed.
#
# it's best we just unload the lighthouse context in binja after
# the UI widgets have been destroyed (which aligns with IDA)
#
if disassembler.NAME == "BINJA":
lctx = self._target.lctx
core = lctx.core
core.binja_close_context(lctx.dctx)
# cleanup the UI / qt references for the CoverageOverview elements
self._target.terminate()
#
# this seems to be 'roughly' the last event triggered after the widget
# is done initializing in both IDA and Binja, but prior to the first
# user-triggered 'show' events.
#
# this is mostly to account for the fact that binja 'shows' the widget
# when it is initially created (outside of our control). this was
# causing lighthouse to automatically cache database metadata when
# every database was opened ...
#
elif int(event.type()) == self.EventLayoutRequest:
self._target.initialized = True
#
# this is used to hook a little bit after the 'show' event of the
# coverage overview. this is the most universal signal that the
# user is *actually* trying to use lighthouse in a meaningful way...
#
# we will use this moment first to check if they skipped straight to
# 'go' and opened the coverage overview without the metadata cache
# getting built.
#
# this case should only happen if the user does 'Show Coverage
# Overview' from the binja-controlled Window menu entry...
#
elif int(event.type()) == self.EventUpdateLater:
if self._target.visible and self._first_hit:
self._first_hit = False
if disassembler.NAME == "BINJA":
self._target.lctx.start()
if not self._target.director.metadata.cached:
self._target.director.refresh()
#
# this is an unknown event, but it seems to fire when the widget is
# being saved/restored by a QMainWidget. we use this to try and ensure
# the Coverage Overview stays docked when flipping between Reversing
# and Debugging states in IDA.
# being saved/restored by a QMainWidget (in IDA). we use this to try
# and ensure the Coverage Overview stays docked when flipping between
# Reversing and Debugging states in IDA.
#
# See issue #16 on github for more information.
#
if int(event.type()) == 2002 and disassembler.NAME == "IDA":
elif int(event.type()) == 2002 and disassembler.NAME == "IDA":
import idaapi
#
@@ -266,11 +371,7 @@ class EventProxy(QtCore.QObject):
# that the user has probably started debugging.
#
# NOTE / COMPAT:
if disassembler.USING_IDA7API:
debug_mode = bool(idaapi.find_widget("General registers"))
else:
debug_mode = bool(idaapi.find_tform("General registers"))
debug_mode = bool(idaapi.find_widget("General registers"))
#
# if this is the first time the user has started debugging, dock
@@ -1,4 +1,5 @@
import logging
from lighthouse.util.qt import *
from lighthouse.util.disassembler import disassembler
@@ -14,8 +15,7 @@ class TableSettingsMenu(QtWidgets.QMenu):
self._visible_action = None
self._ui_init_actions()
if USING_PYQT5:
self.setToolTipsVisible(True)
self.setToolTipsVisible(True)
#--------------------------------------------------------------------------
# QMenu Overloads
@@ -34,19 +34,6 @@ class TableSettingsMenu(QtWidgets.QMenu):
event.accept()
return True
# show action tooltips (for Qt < 5.1)
elif event.type() == QtCore.QEvent.ToolTip and not USING_PYQT5:
if action and self._visible_action != action:
QtWidgets.QToolTip.showText(event.globalPos(), action.toolTip())
self._visible_action = action
event.accept()
return True
# clear tooltips (for Qt < 5.1)
if not (action or USING_PYQT5):
QtWidgets.QToolTip.hideText()
self._visible_action = None
# handle any other events as wee normally should
return super(TableSettingsMenu, self).event(event)
@@ -60,29 +47,28 @@ class TableSettingsMenu(QtWidgets.QMenu):
"""
# lighthouse colors
self._action_colors = QtWidgets.QAction("Colors", None)
self._action_colors.setToolTip("Lighthouse color & theme customization")
#self.addAction(self._action_colors)
#self.addSeparator()
self._action_change_theme = QtWidgets.QAction("Change theme", None)
self._action_change_theme.setToolTip("Lighthouse color & theme customization")
self.addAction(self._action_change_theme)
self.addSeparator()
# painting
self._action_pause_paint = QtWidgets.QAction("Pause painting", None)
self._action_pause_paint.setCheckable(True)
self._action_pause_paint.setToolTip("Disable coverage painting")
self.addAction(self._action_pause_paint)
self._action_force_clear = QtWidgets.QAction("Force clear paint (slow!)", None)
self._action_force_clear.setToolTip("Attempt to forcefully clear stuck paint from the database")
self.addAction(self._action_force_clear)
# misc
self._action_clear_paint = QtWidgets.QAction("Clear paint", None)
self._action_clear_paint.setToolTip("Forcefully clear all paint")
self.addAction(self._action_clear_paint)
self._action_disable_paint = QtWidgets.QAction("Disable painting", None)
self._action_disable_paint.setCheckable(True)
self._action_disable_paint.setToolTip("Disable the coverage painting subsystem")
self.addAction(self._action_disable_paint)
self.addSeparator()
# table actions
self._action_refresh_metadata = QtWidgets.QAction("Full table refresh", None)
self._action_refresh_metadata.setToolTip("Refresh metadata & coverage for db")
self._action_refresh_metadata = QtWidgets.QAction("Rebuild coverage mappings", None)
self._action_refresh_metadata.setToolTip("Refresh the database metadata and coverage mapping")
self.addAction(self._action_refresh_metadata)
self._action_export_html = QtWidgets.QAction("Export to HTML", None)
self._action_export_html = QtWidgets.QAction("Generate HTML report", None)
self._action_export_html.setToolTip("Export the coverage table to HTML")
self.addAction(self._action_export_html)
@@ -91,16 +77,17 @@ class TableSettingsMenu(QtWidgets.QMenu):
self._action_hide_zero.setCheckable(True)
self.addAction(self._action_hide_zero)
def connect_signals(self, controller, core):
def connect_signals(self, controller, lctx):
"""
Connect UI signals.
"""
self._action_refresh_metadata.triggered.connect(controller.refresh_metadata)
self._action_change_theme.triggered.connect(lctx.core.palette.interactive_change_theme)
self._action_refresh_metadata.triggered.connect(lctx.director.refresh)
self._action_hide_zero.triggered[bool].connect(controller._model.filter_zero_coverage)
self._action_pause_paint.triggered[bool].connect(lambda x: core.painter.set_enabled(not x))
self._action_clear_paint.triggered.connect(core.painter.clear_paint)
self._action_disable_paint.triggered[bool].connect(lambda x: lctx.painter.set_enabled(not x))
self._action_force_clear.triggered.connect(lctx.painter.force_clear)
self._action_export_html.triggered.connect(controller.export_to_html)
core.painter.status_changed(self._ui_painter_changed_status)
lctx.painter.status_changed(self._ui_painter_changed_status)
#--------------------------------------------------------------------------
# Signal Handlers
@@ -111,4 +98,4 @@ class TableSettingsMenu(QtWidgets.QMenu):
"""
Handle an event from the painter being enabled/disabled.
"""
self._action_pause_paint.setChecked(not painter_enabled)
self._action_disable_paint.setChecked(not painter_enabled)
@@ -6,6 +6,7 @@ from operator import itemgetter, attrgetter
from lighthouse.util import lmsg
from lighthouse.util.qt import *
from lighthouse.util.python import *
from lighthouse.util.misc import mainthread
from lighthouse.util.disassembler import disassembler
from lighthouse.coverage import FunctionCoverage, BADADDR
@@ -34,6 +35,30 @@ class CoverageTableView(QtWidgets.QTableView):
# configure the widget for use
self._ui_init()
self.refresh_theme()
@disassembler.execute_ui
def refresh_theme(self):
"""
Refresh UI facing elements to reflect the current theme.
"""
palette = self._model.lctx.palette
self.setStyleSheet(
"QTableView {"
" gridline-color: %s;" % palette.table_grid.name() +
" background-color: %s;" % palette.table_background.name() +
" color: %s;" % palette.table_text.name() +
" outline: none; "
"} " +
"QHeaderView::section { "
" padding: 1ex;" \
" margin: 0;" \
"} " +
"QTableView::item:selected {"
" color: white; "
" background-color: %s;" % palette.table_selection.name() +
"}"
)
#--------------------------------------------------------------------------
# QTableView Overloads
@@ -83,22 +108,8 @@ class CoverageTableView(QtWidgets.QTableView):
"""
Initialize the coverage table.
"""
palette = self._model._director._palette
self.setFocusPolicy(QtCore.Qt.StrongFocus)
# widget style
self.setStyleSheet(
"QTableView {"
" gridline-color: black;"
" background-color: %s;" % palette.overview_bg.name() +
#" color: %s;" % palette.combobox_fg.name() +
" outline: none; "
"} " +
"QTableView::item:selected {"
" color: white; "
" background-color: %s;" % palette.selection.name() +
"}"
)
self.setHorizontalScrollMode(QtWidgets.QAbstractItemView.ScrollPerPixel)
# these properties will allow the user shrink the table to any size
self.setMinimumHeight(0)
@@ -119,22 +130,22 @@ class CoverageTableView(QtWidgets.QTableView):
entry_font = self._model.data(0, QtCore.Qt.FontRole)
entry_fm = QtGui.QFontMetricsF(entry_font)
# get the font used by the table cell entries
entry_font = self._model.data(0, QtCore.Qt.FontRole)
entry_fm = QtGui.QFontMetricsF(entry_font)
# set the initial column widths based on their title or contents
for i in xrange(self._model.columnCount()):
# determine the pixel width of the column header text
title_text = self._model.headerData(i, QtCore.Qt.Horizontal)
title_rect = title_fm.boundingRect(title_text)
title_rect = self._model.headerData(i, QtCore.Qt.Horizontal, QtCore.Qt.SizeHintRole)
# determine the pixel width of sample column entry text
entry_text = self._model.SAMPLE_CONTENTS[i]
entry_rect = entry_fm.boundingRect(entry_text)
# select the lager of the two potential column widths
column_width = max(title_rect.width(), entry_rect.width())
# pad the final column width to make the table less dense
column_width = int(column_width * 1.2)
# select the larger of the two potential column widths
column_width = int(max(title_rect.width(), entry_rect.width()*1.2))
# save the final column width
self.setColumnWidth(i, column_width)
@@ -154,7 +165,9 @@ class CoverageTableView(QtWidgets.QTableView):
vh.hide()
# stretch last table column (which is blank) to fill remaining space
hh.setStretchLastSection(True)
#hh.setStretchLastSection(True)
#hh.setCascadingSectionResizes(True)
hh.setSectionResizeMode(1, QtWidgets.QHeaderView.Stretch)
# disable bolding of table column headers when table is selected
hh.setHighlightSections(False)
@@ -172,26 +185,23 @@ class CoverageTableView(QtWidgets.QTableView):
#
# force the table row heights to be fixed height
if USING_PYQT5:
vh.setSectionResizeMode(QtWidgets.QHeaderView.Fixed)
else:
vh.setResizeMode(QtWidgets.QHeaderView.Fixed)
# specify the fixed pixel height for the table headers
spacing = title_fm.height() - title_fm.xHeight()
tweak = 24 - spacing
hh.setFixedHeight(entry_fm.height()+tweak)
vh.setSectionResizeMode(QtWidgets.QHeaderView.Fixed)
# specify the fixed pixel height for the table rows
# NOTE: don't ask too many questions about this voodoo math :D
spacing = entry_fm.height() - entry_fm.xHeight()
tweak = 16 - spacing
vh.setDefaultSectionSize(entry_fm.height()+tweak)
tweak = (17*get_dpi_scale() - spacing)/get_dpi_scale()
vh.setDefaultSectionSize(int(entry_fm.height()+tweak))
def _ui_init_table_ctx_menu_actions(self):
"""
Initialize the right click context menu actions for the table view.
"""
# misc actions
self._action_dump_orphan = QtWidgets.QAction("Dump orphan addresses", None)
self._action_dump_internal = QtWidgets.QAction("Dump internal addresses (Debug)", None)
# function actions
self._action_rename = QtWidgets.QAction("Rename", None)
self._action_copy_name = QtWidgets.QAction("Copy name", None)
@@ -255,7 +265,12 @@ class CoverageTableView(QtWidgets.QTableView):
return
# show the popup menu to the user, and wait for their selection
action = ctx_menu.exec_(self.viewport().mapToGlobal(position))
if USING_PYSIDE6:
exec_func = getattr(ctx_menu, "exec")
else:
exec_func = getattr(ctx_menu, "exec_")
action = exec_func(self.viewport().mapToGlobal(position))
# process the user action
self._process_table_ctx_menu_action(action)
@@ -275,7 +290,11 @@ class CoverageTableView(QtWidgets.QTableView):
return
# show the popup menu to the user, and wait for their selection
action = ctx_menu.exec_(hh.viewport().mapToGlobal(position))
if USING_PYSIDE6:
exec_func = getattr(ctx_menu, "exec")
else:
exec_func = getattr(ctx_menu, "exec_")
action = exec_func(hh.viewport().mapToGlobal(position))
# process the user action
self._process_header_ctx_menu_action(action, column)
@@ -292,8 +311,8 @@ class CoverageTableView(QtWidgets.QTableView):
"""
# get the list rows currently selected in the coverage table
selected_rows = self.selectionModel().selectedRows()
if len(selected_rows) == 0:
selected_row_indexes = self.selectionModel().selectedRows()
if len(selected_row_indexes) == 0:
return None
# the context menu we will dynamically populate
@@ -305,13 +324,25 @@ class CoverageTableView(QtWidgets.QTableView):
# copy function name, address, or renaming the function.
#
if len(selected_rows) == 1:
ctx_menu.addAction(self._action_rename)
ctx_menu.addSeparator()
ctx_menu.addAction(self._action_copy_name)
ctx_menu.addAction(self._action_copy_address)
ctx_menu.addAction(self._action_copy_name_and_address)
ctx_menu.addSeparator()
if len(selected_row_indexes) == 1:
row = selected_row_indexes[0].row()
function_address = self._model.row2func[row]
# special handling for right click of orphan coverage row
if function_address == BADADDR:
ctx_menu.addAction(self._action_dump_orphan)
ctx_menu.addAction(self._action_dump_internal)
return ctx_menu
# normal right click of a function row
else:
ctx_menu.addAction(self._action_rename)
ctx_menu.addSeparator()
ctx_menu.addAction(self._action_copy_name)
ctx_menu.addAction(self._action_copy_address)
ctx_menu.addAction(self._action_copy_name_and_address)
ctx_menu.addSeparator()
#
# if multiple functions are selected then show actions available
@@ -370,6 +401,14 @@ class CoverageTableView(QtWidgets.QTableView):
elif action == self._action_clear_prefix:
self._controller.clear_function_prefixes(rows)
# handle the 'Dump orphan addresses' action
elif action == self._action_dump_orphan:
self._controller.dump_orphan()
# handle the 'Dump internal addresses' action
elif action == self._action_dump_internal:
self._controller.dump_internal()
#--------------------------------------------------------------------------
# Context Menu (Table Header)
#--------------------------------------------------------------------------
@@ -406,7 +445,8 @@ class CoverageTableController(object):
The Coverage Table Controller (Logic)
"""
def __init__(self, model):
def __init__(self, lctx, model):
self.lctx = lctx
self._model = model
self._last_directory = None
@@ -422,7 +462,10 @@ class CoverageTableController(object):
# retrieve details about the function targeted for rename
function_address = self._model.row2func[row]
original_name = disassembler.get_function_raw_name_at(function_address)
if function_address == BADADDR:
return
original_name = disassembler[self.lctx].get_function_raw_name_at(function_address)
# prompt the user for a new function name
ok, new_name = prompt_string(
@@ -440,7 +483,7 @@ class CoverageTableController(object):
return
# rename the function
disassembler.set_function_name_at(function_address, new_name)
disassembler[self.lctx].set_function_name_at(function_address, new_name)
@mainthread
def prefix_table_functions(self, rows):
@@ -461,7 +504,7 @@ class CoverageTableController(object):
# apply the user prefix to the functions depicted in the given rows
function_addresses = self._get_function_addresses(rows)
disassembler.prefix_functions(function_addresses, prefix)
disassembler[self.lctx].prefix_functions(function_addresses, prefix)
@mainthread
def clear_function_prefixes(self, rows):
@@ -469,7 +512,7 @@ class CoverageTableController(object):
Clear prefixes of database functions via the coverage table.
"""
function_addresses = self._get_function_addresses(rows)
disassembler.clear_prefixes(function_addresses)
disassembler[self.lctx].clear_prefixes(function_addresses)
#---------------------------------------------------------------------------
# Copy-to-Clipboard
@@ -520,6 +563,38 @@ class CoverageTableController(object):
copy_to_clipboard(function_name_and_address.rstrip())
return function_name_and_address
#--------------------------------------------------------------------------
# Dumping
#--------------------------------------------------------------------------
def dump_orphan(self):
"""
Dump the orphan coverage data.
"""
coverage = self.lctx.director.coverage
lmsg("Orphan coverage addresses for %s:" % coverage.name)
self._dump_addresses(coverage.orphan_addresses)
def dump_internal(self):
"""
Dump the internal coverage data.
"""
coverage = self.lctx.director.coverage
lmsg("Internal coverage addresses for %s:" % coverage.name)
self._dump_addresses(coverage.unmapped_addresses)
def _dump_addresses(self, coverage_addresses):
"""
Dump the given list of addresses to the terminal.
"""
coverage_addresses = sorted(coverage_addresses)
if not coverage_addresses:
lmsg(" * (there is no addresses to dump)")
return
for address in coverage_addresses:
lmsg(" * 0x%X" % address)
#---------------------------------------------------------------------------
# Misc
#---------------------------------------------------------------------------
@@ -528,7 +603,34 @@ class CoverageTableController(object):
"""
Navigate to the function depicted by the given row.
"""
disassembler.navigate(self._model.row2func[row])
# get the clicked function address
function_address = self._model.row2func[row]
if function_address == BADADDR:
return
#
# if there is actually coverage in the function, attempt to locate the
# first block (or any block) with coverage and set that as our target
#
function_coverage = self.lctx.director.coverage.functions.get(function_address, None)
if function_coverage:
if function_address in function_coverage.nodes:
target_address = function_address
else:
target_address = sorted(function_coverage.nodes)[0]
#
# if the user clicked a function with no coverage, we should just
# navigate to the top of the function... nothing fancy
#
else:
target_address = function_address
# navigate to the target function + block
disassembler[self.lctx].navigate_to_function(function_address, target_address)
def toggle_column_alignment(self, column):
"""
@@ -539,36 +641,22 @@ class CoverageTableController(object):
# toggle the column alignment between center (default) and left
if alignment == QtCore.Qt.AlignCenter:
new_alignment = QtCore.Qt.AlignLeft | QtCore.Qt.AlignVCenter
new_alignment = QtCore.Qt.AlignVCenter
else:
new_alignment = QtCore.Qt.AlignCenter
# send the new alignment to the model
self._model.set_column_alignment(column, new_alignment)
def refresh_metadata(self):
"""
Hard refresh of the director and table metadata layers.
"""
disassembler.show_wait_box("Building database metadata...")
self._model._director.refresh()
# ensure the table's model gets refreshed
disassembler.replace_wait_box("Refreshing Coverage Overview...")
self._model.refresh()
# all done
disassembler.hide_wait_box()
def export_to_html(self):
"""
Export the coverage table to an HTML report.
"""
if not self._last_directory:
self._last_directory = disassembler.get_database_directory()
self._last_directory = disassembler[self.lctx].get_database_directory()
# build filename for the coverage report based off the coverage name
name, _ = os.path.splitext(self._model._director.coverage_name)
name, _ = os.path.splitext(self.lctx.director.coverage_name)
filename = name + ".html"
suggested_filepath = os.path.join(self._last_directory, filename)
@@ -581,9 +669,13 @@ class CoverageTableController(object):
{
"filter": "HTML Files (*.html)",
"caption": "Save HTML Report",
"directory" if USING_PYQT5 else "dir": suggested_filepath
}
if USING_PYQT5:
kwargs["directory"] = suggested_filepath
else:
kwargs["dir"] = suggested_filepath
# prompt the user with the file dialog, and await their chosen filename(s)
filename, _ = file_dialog.getSaveFileName(**kwargs)
if not filename:
@@ -593,7 +685,7 @@ class CoverageTableController(object):
self._last_directory = os.path.dirname(filename) + os.sep
# write the generated HTML report to disk
with open(filename, "wb") as fd:
with open(filename, "w") as fd:
fd.write(self._model.to_html())
lmsg("Saved HTML report to %s" % filename)
@@ -609,7 +701,8 @@ class CoverageTableController(object):
function_addresses = []
for row_number in rows:
address = self._model.row2func[row_number]
function_addresses.append(address)
if address != BADADDR:
function_addresses.append(address)
return function_addresses
#------------------------------------------------------------------------------
@@ -629,7 +722,6 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
INST_HIT = 4
FUNC_SIZE = 5
COMPLEXITY = 6
FINAL_COLUMN = 7
METADATA_ATTRIBUTES = [FUNC_NAME, FUNC_ADDR, FUNC_SIZE, COMPLEXITY]
COVERAGE_ATTRIBUTES = [COV_PERCENT, BLOCKS_HIT, INST_HIT]
@@ -649,32 +741,43 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
# column headers of the table
COLUMN_HEADERS = \
{
COV_PERCENT: "Coverage %",
FUNC_NAME: "Function Name",
COV_PERCENT: "Cov %",
FUNC_NAME: "Func Name",
FUNC_ADDR: "Address",
BLOCKS_HIT: "Blocks Hit",
INST_HIT: "Instructions Hit",
FUNC_SIZE: "Function Size",
COMPLEXITY: "Complexity",
FINAL_COLUMN: ""
INST_HIT: "Instr. Hit",
FUNC_SIZE: "Func Size",
COMPLEXITY: "CC",
}
# column header tooltips
COLUMN_TOOLTIPS = \
{
COV_PERCENT: "Coverage Percent",
FUNC_NAME: "Function Name",
FUNC_ADDR: "Function Address",
BLOCKS_HIT: "Number of Basic Blocks Executed",
INST_HIT: "Number of Instructions Executed",
FUNC_SIZE: "Function Size (bytes)",
COMPLEXITY: "Cyclomatic Complexity",
}
# sample column
SAMPLE_CONTENTS = \
[
" 100.00% ",
" 100.00 ",
" sub_140001B20 ",
" 0x140001b20 ",
" 100 / 100 ",
" 1000 / 1000 ",
" 10000000 ",
" 1000000 ",
""
" 100000 ",
" 1000 ",
]
def __init__(self, director, parent=None):
def __init__(self, lctx, parent=None):
super(CoverageTableModel, self).__init__(parent)
self._director = director
self.lctx = lctx
self._director = lctx.director
# convenience mapping from row_number --> function_address
self.row2func = {}
@@ -687,7 +790,7 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
# a fallback coverage object for functions with no coverage
self._blank_coverage = FunctionCoverage(BADADDR)
self._blank_coverage.coverage_color = director._palette.coverage_none
self._blank_coverage.coverage_color = lctx.palette.table_coverage_none
# set the default column text alignment for each column (centered)
self._default_alignment = QtCore.Qt.AlignCenter
@@ -695,14 +798,19 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
self._default_alignment for x in self.COLUMN_HEADERS
]
# make the function name column left aligned by default
self.set_column_alignment(self.FUNC_NAME, QtCore.Qt.AlignVCenter)
# initialize a monospace font to use for table row / cell text
self._entry_font = MonospaceFont()
self._entry_font.setStyleStrategy(QtGui.QFont.ForceIntegerMetrics)
self._entry_font.setPointSizeF(normalize_to_dpi(9))
if not USING_PYSIDE6:
#TODO Figure out if this matters?
self._entry_font.setStyleStrategy(QtGui.QFont.ForceIntegerMetrics)
self._entry_font.setPointSizeF(normalize_to_dpi(10))
# use the default / system font for the column titles
self._title_font = QtGui.QFont()
self._title_font.setPointSizeF(normalize_to_dpi(9))
self._title_font.setPointSizeF(normalize_to_dpi(10))
#----------------------------------------------------------------------
# Sorting
@@ -729,7 +837,16 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
# register for cues from the director
self._director.coverage_switched(self._internal_refresh)
self._director.coverage_modified(self._internal_refresh)
self._director.metadata_modified(self._data_changed)
self._director.metadata.function_renamed(self._data_changed)
def refresh_theme(self):
"""
Refresh UI facing elements to reflect the current theme.
Does not require @disassembler.execute_ui decorator, data_changed() has its own.
"""
self._blank_coverage.coverage_color = self.lctx.palette.table_coverage_none
self._data_changed()
#--------------------------------------------------------------------------
# QAbstractTableModel Overloads
@@ -767,13 +884,104 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
# center align all columns
return self._column_alignment[column]
# tooltip request
elif role == QtCore.Qt.ToolTipRole:
return self.COLUMN_TOOLTIPS[column]
# font format request
elif role == QtCore.Qt.FontRole:
return self._title_font
if role == QtCore.Qt.SizeHintRole:
title_fm = QtGui.QFontMetricsF(self._title_font)
title_rect = title_fm.boundingRect(self.COLUMN_HEADERS[column])
padded = QtCore.QSize(int(title_rect.width()*1.45), int(title_rect.height()*1.75))
return padded
# unhandeled header request
return None
def _data_function_display(self, function_address, column):
"""
Return a string to diplay in the requested column of the given function.
"""
# lookup the function info for the given function
try:
function_metadata = self.lctx.metadata.functions[function_address]
#
# if we hit a KeyError, it is probably because the database metadata
# is being refreshed and the model (this object) has yet to be
# updated.
#
# this should only ever happen as a result of the user using the
# right click 'Refresh metadata' action. And even then, only when
# a function they undefined in the IDB is visible in the coverage
# overview table view.
#
# In theory, the table should get refreshed *after* the metadata
# refresh completes. So for now, we simply return return the filler
# string '?'
#
except KeyError:
return "?"
#
# remember, if a function does *not* have coverage data, it will
# not have an entry in the coverage map. that means we should
# yield a default, 'blank', coverage item in these instances
#
function_coverage = self._director.coverage.functions.get(
function_address,
self._blank_coverage
)
# Coverage % - (by instruction execution)
if column == self.COV_PERCENT:
return "%5.2f" % (function_coverage.instruction_percent*100)
# Function Name
elif column == self.FUNC_NAME:
return function_metadata.name
# Function Address
elif column == self.FUNC_ADDR:
return "0x%X" % function_metadata.address
# Basic Blocks
elif column == self.BLOCKS_HIT:
return "%3u / %-3u" % (function_coverage.nodes_executed,
function_metadata.node_count)
# Instructions Hit
elif column == self.INST_HIT:
return "%4u / %-4u" % (function_coverage.instructions_executed,
function_metadata.instruction_count)
# Function Size
elif column == self.FUNC_SIZE:
return "%u" % function_metadata.size
# Cyclomatic Complexity
elif column == self.COMPLEXITY:
return "%u" % function_metadata.cyclomatic_complexity
# unhandeled? maybe make this an assert?
return None
def _data_orphan_display(self, column):
"""
Return a string to be displayed by the table
"""
if column == self.FUNC_NAME:
return "Orphan Coverage"
elif column == self.INST_HIT:
return "%u" % len(self._director.coverage.orphan_addresses)
return "N/A"
def data(self, index, role=QtCore.Qt.DisplayRole):
"""
Define how Qt should access the underlying model data.
@@ -782,85 +990,32 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
# a request has been made for what text to show in a table cell
if role == QtCore.Qt.DisplayRole:
# alias the requested column number once, for readability & perf
column = index.column()
function_address = self.row2func[index.row()]
# lookup the function info for this row
try:
function_address = self.row2func[index.row()]
function_metadata = self._director.metadata.functions[function_address]
#
# if we hit a KeyError, it is probably because the database metadata
# is being refreshed and the model (this object) has yet to be
# updated.
#
# this should only ever happen as a result of the user using the
# right click 'Refresh metadata' action. And even then, only when
# a function they undefined in the IDB is visible in the coverage
# overview table view.
#
# In theory, the table should get refreshed *after* the metadata
# refresh completes. So for now, we simply return return the filler
# string '?'
#
except KeyError:
return "?"
#
# remember, if a function does *not* have coverage data, it will
# not have an entry in the coverage map. that means we should
# yield a default, 'blank', coverage item in these instances
#
function_coverage = self._director.coverage.functions.get(
function_address,
self._blank_coverage
)
# Coverage % - (by instruction execution)
if column == self.COV_PERCENT:
return "%5.2f" % (function_coverage.instruction_percent*100)
# Function Name
elif column == self.FUNC_NAME:
return function_metadata.name
# Function Address
elif column == self.FUNC_ADDR:
return "0x%X" % function_metadata.address
# Basic Blocks
elif column == self.BLOCKS_HIT:
return "%3u / %-3u" % (function_coverage.nodes_executed,
function_metadata.node_count)
# Instructions Hit
elif column == self.INST_HIT:
return "%4u / %-4u" % (function_coverage.instructions_executed,
function_metadata.instruction_count)
# Function Size
elif column == self.FUNC_SIZE:
return "%u" % function_metadata.size
# Cyclomatic Complexity
elif column == self.COMPLEXITY:
return "%u" % function_metadata.cyclomatic_complexity
if function_address == BADADDR:
return self._data_orphan_display(column)
else:
return self._data_function_display(function_address, column)
# cell background color request
elif role == QtCore.Qt.BackgroundRole:
function_address = self.row2func[index.row()]
# special handling for 'orphan' coverage
if function_address == BADADDR:
# if there was *ANY* coverage, color the 'orphan' line red
if self._director.coverage.orphan_addresses:
return self.lctx.palette.table_coverage_bad
# normal handling
function_coverage = self._director.coverage.functions.get(
function_address,
self._blank_coverage
)
return function_coverage.coverage_color
# cell text color request
elif role == QtCore.Qt.ForegroundRole:
return QtGui.QColor(QtCore.Qt.white)
return function_coverage.coverage_color
# cell font style format request
elif role == QtCore.Qt.FontRole:
@@ -892,10 +1047,17 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
# column has not been enlightened to sorting
except KeyError as e:
logger.warning("TODO/FUTURE: implement column %u sorting?" % column)
logger.error("ERROR: Sorting not implemented for column %u" % column)
self.layoutChanged.emit()
return
#
# In PySide6 (eg. Binary Ninja) the Qt.SortOrder type does not convert
# to a simple integer (unlike PyQt5) so we reduce the type for comapt
#
direction = (sort_order == QtCore.Qt.SortOrder.DescendingOrder)
#
# NOTE: attrgetter appears to profile ~8-12% faster than lambdas
# accessing the member on the member, hence the strange paradigm
@@ -904,17 +1066,17 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
# sort the table entries by a function metadata attribute
if column in self.METADATA_ATTRIBUTES:
sorted_functions = sorted(
self._visible_metadata.itervalues(),
itervalues(self._visible_metadata),
key=attrgetter(sort_field),
reverse=sort_order
reverse=direction
)
# sort the table entries by a function coverage attribute
elif column in self.COVERAGE_ATTRIBUTES:
sorted_functions = sorted(
self._visible_coverage.itervalues(),
itervalues(self._visible_coverage),
key=attrgetter(sort_field),
reverse=sort_order
reverse=direction
)
#
@@ -932,7 +1094,7 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
# items (0%) should be appended to the *end*
#
if sort_order:
if direction:
sorted_functions += self._no_coverage
#
@@ -948,7 +1110,8 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
# finally, rebuild the row2func mapping and notify views of this change
self.row2func = dict(zip(xrange(len(sorted_functions)), sorted_addresses))
self.func2row = {v: k for k, v in self.row2func.iteritems()}
self.row2func[len(self.row2func)] = BADADDR
self.func2row = {v: k for k, v in iteritems(self.row2func)}
self.layoutChanged.emit()
# save the details of this sort event as they may be needed later
@@ -975,12 +1138,12 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
# sum the # of instructions in all the visible functions
instruction_count = sum(
meta.instruction_count for meta in self._visible_metadata.itervalues()
meta.instruction_count for meta in itervalues(self._visible_metadata)
)
# sum the # of instructions executed in all the visible functions
instructions_executed = sum(
cov.instructions_executed for cov in self._visible_coverage.itervalues()
cov.instructions_executed for cov in itervalues(self._visible_coverage)
)
# compute coverage percentage of the visible functions
@@ -994,6 +1157,7 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
"""
Generate an HTML representation of the coverage table.
"""
palette = self.lctx.palette
# table summary
summary_html, summary_css = self._generate_html_summary()
@@ -1006,13 +1170,16 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
body_html = "<body>%s</body>" % '\n'.join(body_elements)
body_css = \
"""
body {
body {{
font-family: Arial, Helvetica, sans-serif;
color: white;
background-color: #363636;
}
"""
color: {page_fg};
background-color: {page_bg};
}}
""".format(
page_fg=palette.table_text.name(),
page_bg=palette.html_page_background.name()
)
# HTML <head> tag
css_elements = [body_css, summary_css, table_css]
@@ -1030,6 +1197,7 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
"""
Generate the HTML table summary.
"""
palette = self.lctx.palette
metadata = self._director.metadata
coverage = self._director.coverage
@@ -1052,9 +1220,17 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
list_html = "<ul>%s</ul>" % '\n'.join(details)
list_css = \
"""
.detail { font-weight: bold; color: white; }
li { color: #c0c0c0; }
"""
.detail {{
font-weight: bold;
color: {page_fg};
}}
li {{
color: {detail_fg};
}}
""".format(
page_fg=palette.table_text.name(),
detail_fg=palette.html_summary_text.name()
)
# title + summary
summary_html = title_html + list_html
@@ -1065,7 +1241,7 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
"""
Generate the HTML coverage table.
"""
palette = self._director._palette
palette = self.lctx.palette
table_rows = []
# generate the table's column title row
@@ -1074,7 +1250,7 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
header_cells.append(
"<th>%s</th>" % self.headerData(i, QtCore.Qt.Horizontal)
)
table_rows.append(("#505050", header_cells))
table_rows.append((palette.html_table_header.name(), header_cells))
# generate the table's coverage rows
for row in xrange(self.rowCount()):
@@ -1122,8 +1298,8 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
padding: 1ex 1em 1ex 1em;
}}
""".format(
table_bg=palette.overview_bg.name(),
table_fg="white"
table_bg=palette.table_background.name(),
table_fg=palette.table_text.name()
)
return (table_html, table_css)
@@ -1204,7 +1380,7 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
normalize = lambda x: x
if not (set(self._search_string) & set(string.ascii_uppercase)):
normalize = lambda x: string.lower(x)
normalize = lambda x: x.lower()
#
# it's time to rebuild the list of coverage items to make visible in
@@ -1213,7 +1389,7 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
#
# loop through *all* the functions as defined in the active metadata
for function_address in metadata.functions.iterkeys():
for function_address in metadata.functions:
#------------------------------------------------------------------
# Filters - START
@@ -1246,8 +1422,11 @@ class CoverageTableModel(QtCore.QAbstractTableModel):
self.row2func[row] = function_address
row += 1
# add a special entry for 'orphan coverage'
self.row2func[len(self.row2func)] = BADADDR
# build the inverse func --> row mapping
self.func2row = {v: k for k, v in self.row2func.iteritems()}
self.func2row = {v: k for k, v in iteritems(self.row2func)}
# bake the final number of rows into the model
self._row_count = len(self.row2func)
+176
View File
@@ -0,0 +1,176 @@
import os
import logging
from lighthouse.util import lmsg
from lighthouse.util.qt import *
from lighthouse.util.misc import human_timestamp
from lighthouse.util.python import *
logger = logging.getLogger("Lighthouse.UI.Xref")
#------------------------------------------------------------------------------
# Coverage Xref Dialog
#------------------------------------------------------------------------------
class CoverageXref(QtWidgets.QDialog):
"""
A Qt Dialog to list other coverage sets that contain a given address.
This class makes up a rudimentary xref dialog. It does not follow Qt
'best practices' because it does not need to be super flashy, nor does
it demand much facetime.
"""
def __init__(self, director, address):
super(CoverageXref, self).__init__()
self._director = director
# dialog attributes
self.address = address
self.selected_coverage = None
self.selected_filepath = None
# configure the widget for use
self._ui_init()
#--------------------------------------------------------------------------
# Initialization - UI
#--------------------------------------------------------------------------
def _ui_init(self):
"""
Initialize UI elements.
"""
self.setWindowTitle("Coverage Xrefs to 0x%X" % self.address)
self.setWindowFlags(self.windowFlags() & ~QtCore.Qt.WindowContextHelpButtonHint)
self.setModal(True)
self._font = self.font()
self._font.setPointSizeF(normalize_to_dpi(10))
self._font_metrics = QtGui.QFontMetricsF(self._font)
# initialize coverage xref table
self._ui_init_table()
self._populate_table()
# layout the populated UI just before showing it
self._ui_layout()
def _ui_init_table(self):
"""
Initialize the coverage xref table UI elements.
"""
self._table = QtWidgets.QTableWidget()
self._table.verticalHeader().setVisible(False)
self._table.setHorizontalScrollMode(QtWidgets.QAbstractItemView.ScrollPerPixel)
self._table.horizontalHeader().setFont(self._font)
self._table.setFont(self._font)
self._table.setWordWrap(False)
# symbol, cov %, name, time
self._table.setColumnCount(4)
self._table.setHorizontalHeaderLabels(["Sym", "Cov %", "Coverage Name", "Timestamp"])
self._table.setColumnWidth(0, 45)
self._table.setColumnWidth(1, 55)
self._table.setColumnWidth(2, 400)
self._table.setColumnWidth(3, 100)
# left align text in column headers
for i in range(4):
self._table.horizontalHeaderItem(i).setTextAlignment(QtCore.Qt.AlignLeft)
# disable bolding of column headers when selected
self._table.horizontalHeader().setHighlightSections(False)
# stretch the filename field, as it is the most important
self._table.horizontalHeader().setSectionResizeMode(2, QtWidgets.QHeaderView.Stretch)
# make table read only, select a full row by default
self._table.setEditTriggers(QtWidgets.QAbstractItemView.NoEditTriggers)
self._table.setSelectionBehavior(QtWidgets.QAbstractItemView.SelectRows)
# catch double click events on table rows
self._table.cellDoubleClicked.connect(self._ui_cell_double_click)
def _populate_table(self):
"""
Populate the xref table with data from the coverage director.
"""
cov_xrefs = self._director.get_address_coverage(self.address)
file_xrefs = self._director.get_address_file(self.address)
# dedupe
for coverage in cov_xrefs:
if coverage.filepath in file_xrefs:
file_xrefs.remove(coverage.filepath)
# populate table with coverage details
self._table.setSortingEnabled(False)
self._table.setRowCount(len(cov_xrefs) + len(file_xrefs))
# coverage objects
for i, coverage in enumerate(cov_xrefs, 0):
self._table.setItem(i, 0, QtWidgets.QTableWidgetItem(self._director.get_shorthand(coverage.name)))
self._table.setItem(i, 1, QtWidgets.QTableWidgetItem("%5.2f" % (coverage.instruction_percent*100)))
name_entry = QtWidgets.QTableWidgetItem(coverage.name)
name_entry.setToolTip(coverage.filepath)
self._table.setItem(i, 2, name_entry)
date_entry = QtWidgets.QTableWidgetItem()
date_entry.setData(QtCore.Qt.DisplayRole, QtCore.QDateTime.fromMSecsSinceEpoch(int(coverage.timestamp*1000)))
self._table.setItem(i, 3, QtWidgets.QTableWidgetItem(date_entry))
# filepaths
for i, filepath in enumerate(file_xrefs, len(cov_xrefs)):
# try to read timestamp of the file on disk (if it exists)
try:
timestamp = os.path.getmtime(filepath)
except (OSError, TypeError):
timestamp = 0
# populate table entry
self._table.setItem(i, 0, QtWidgets.QTableWidgetItem("-"))
self._table.setItem(i, 1, QtWidgets.QTableWidgetItem("-"))
name_entry = QtWidgets.QTableWidgetItem(os.path.basename(filepath))
name_entry.setToolTip(filepath)
self._table.setItem(i, 2, name_entry)
date_entry = QtWidgets.QTableWidgetItem()
date_entry.setData(QtCore.Qt.DisplayRole, QtCore.QDateTime.fromMSecsSinceEpoch(int(timestamp*1000)))
self._table.setItem(i, 3, date_entry)
self._table.resizeColumnsToContents()
self._table.resizeRowsToContents()
self._table.setSortingEnabled(True)
def _ui_layout(self):
"""
Layout the major UI elements of the widget.
"""
layout = QtWidgets.QVBoxLayout()
# layout child widgets
layout.addWidget(self._table)
# scale widget dimensions based on DPI
height = int(get_dpi_scale() * 250)
width = int(get_dpi_scale() * 600)
self.setMinimumHeight(height)
self.setMinimumWidth(width)
# apply the widget layout
self.setLayout(layout)
#--------------------------------------------------------------------------
# Signal Handlers
#--------------------------------------------------------------------------
def _ui_cell_double_click(self, row, column):
"""
A cell/row has been double clicked in the xref table.
"""
if self._table.item(row, 0).text() == "-":
self.selected_filepath = self._table.item(row, 2).toolTip()
else:
self.selected_coverage = self._table.item(row, 2).text()
self.accept()
+168
View File
@@ -0,0 +1,168 @@
import os
import logging
from lighthouse.util import lmsg
from lighthouse.util.qt import *
from lighthouse.util.misc import human_timestamp
from lighthouse.util.python import *
logger = logging.getLogger("Lighthouse.UI.ModuleSelector")
#------------------------------------------------------------------------------
# Coverage Xref Dialog
#------------------------------------------------------------------------------
class ModuleSelector(QtWidgets.QDialog):
"""
A Qt Dialog to list all the coverage modules in a coverage file.
This class makes up a rudimentary selector dialog. It does not follow Qt
'best practices' because it does not need to be super flashy, nor does
it demand much facetime.
"""
def __init__(self, target_name, module_names, coverage_file):
super(ModuleSelector, self).__init__()
self._target_name = target_name
self._module_names = module_names
self._coverage_file = os.path.basename(coverage_file)
# dialog attributes
self.selected_name = None
# configure the widget for use
self._ui_init()
@property
def remember_alias(self):
return self._checkbox_remember.isChecked()
@property
def ignore_missing(self):
return self._checkbox_ignore_missing.isChecked()
#--------------------------------------------------------------------------
# Initialization - UI
#--------------------------------------------------------------------------
def _ui_init(self):
"""
Initialize UI elements.
"""
self.setWindowTitle("Select module matching this database")
self.setWindowFlags(self.windowFlags() & ~QtCore.Qt.WindowContextHelpButtonHint)
self.setModal(True)
self._font = self.font()
self._font.setPointSizeF(normalize_to_dpi(10))
self._font_metrics = QtGui.QFontMetricsF(self._font)
# initialize module selector table
self._ui_init_header()
self._ui_init_table()
self._populate_table()
# layout the populated UI just before showing it
self._ui_layout()
def _ui_init_header(self):
"""
Initialize the module selector header UI elements.
"""
description_text = \
"Lighthouse could not automatically identify the target module in the given coverage file:<br />" \
"<br />" \
"-- <b>Target:</b> {0}<br />" \
"-- <b>Coverage File:</b> {1}<br />" \
"<br />" \
"Please double click the name of the module that matches this database, or close this dialog<br />" \
"if you do not see your binary listed in the table below...".format(self._target_name, self._coverage_file)
self._label_description = QtWidgets.QLabel(description_text)
self._label_description.setTextFormat(QtCore.Qt.RichText)
self._label_description.setFont(self._font)
#self._label_description.setWordWrap(True)
# a checkbox to save the user selected alias to the database
self._checkbox_remember = QtWidgets.QCheckBox("Remember target module alias for this session")
self._checkbox_remember.setFont(self._font)
# a checkbox to ignore future 'missing coverage' / select module warnings
self._checkbox_ignore_missing = QtWidgets.QCheckBox("Suppress this dialog for the remaining coverage files")
self._checkbox_ignore_missing.setFont(self._font)
def _ui_init_table(self):
"""
Initialize the module selector table UI elements.
"""
self._table = QtWidgets.QTableWidget()
self._table.verticalHeader().setVisible(False)
self._table.setHorizontalScrollMode(QtWidgets.QAbstractItemView.ScrollPerPixel)
self._table.horizontalHeader().setFont(self._font)
self._table.setFont(self._font)
# Create a simple table / list
self._table.setColumnCount(1)
self._table.setHorizontalHeaderLabels(["Module Name"])
# left align text in column headers
self._table.horizontalHeaderItem(0).setTextAlignment(QtCore.Qt.AlignLeft)
# disable bolding of column headers when selected
self._table.horizontalHeader().setHighlightSections(False)
# stretch the last column of the table (aesthetics)
self._table.horizontalHeader().setStretchLastSection(True)
# make table read only, select a full row by default
self._table.setEditTriggers(QtWidgets.QAbstractItemView.NoEditTriggers)
self._table.setSelectionBehavior(QtWidgets.QAbstractItemView.SelectRows)
# catch double click events on table rows
self._table.cellDoubleClicked.connect(self._ui_cell_double_click)
def _populate_table(self):
"""
Populate the module table with the module names provided to this dialog.
"""
self._table.setSortingEnabled(False)
self._table.setRowCount(len(self._module_names))
for i, module_name in enumerate(self._module_names, 0):
self._table.setItem(i, 0, QtWidgets.QTableWidgetItem(module_name))
self._table.resizeRowsToContents()
self._table.setSortingEnabled(True)
def _ui_layout(self):
"""
Layout the major UI elements of the widget.
"""
layout = QtWidgets.QVBoxLayout()
#layout.setContentsMargins(0,0,0,0)
# layout child widgets
layout.addWidget(self._label_description)
layout.addWidget(self._table)
layout.addWidget(self._checkbox_remember)
layout.addWidget(self._checkbox_ignore_missing)
# scale widget dimensions based on DPI
height = int(get_dpi_scale() * 250)
width = int(get_dpi_scale() * 400)
self.setMinimumHeight(height)
self.setMinimumWidth(width)
# apply the widget layout
self.setLayout(layout)
#--------------------------------------------------------------------------
# Signal Handlers
#--------------------------------------------------------------------------
def _ui_cell_double_click(self, row, column):
"""
A cell/row has been double clicked in the module table.
"""
self.selected_name = self._table.item(row, 0).text()
self.accept()
+573
View File
@@ -0,0 +1,573 @@
import os
import json
import glob
import shutil
import logging
# NOTE: Py2/Py3 compat
try:
from json.decoder import JSONDecodeError
except ImportError:
JSONDecodeError = ValueError
from lighthouse.util.qt import *
from lighthouse.util.log import lmsg
from lighthouse.util.misc import *
from lighthouse.util.disassembler import disassembler
logger = logging.getLogger("Lighthouse.UI.Palette")
#------------------------------------------------------------------------------
# Plugin Color Palette
#------------------------------------------------------------------------------
class LighthousePalette(object):
"""
Color Palette for the Lighthouse plugin.
"""
def __init__(self):
"""
Initialize default palette colors for Lighthouse.
"""
self._initialized = False
self._last_directory = None
self._required_fields = []
# hints about the user theme (light/dark)
self._user_qt_hint = "dark"
self._user_disassembly_hint = "dark"
self.theme = None
self._default_themes = \
{
"dark": "synth.json",
"light": "dullien.json"
}
# list of objects requesting a callback after a theme change
self._theme_changed_callbacks = []
# get a list of required theme fields, for user theme validation
self._load_required_fields()
# initialize the user theme directory
self._populate_user_theme_dir()
# load a placeholder theme (unhinted) for inital Lighthoue bring-up
self._load_preferred_theme(True)
self._initialized = False
@staticmethod
def get_plugin_theme_dir():
"""
Return the Lighthouse plugin theme directory.
"""
return plugin_resource("themes")
@staticmethod
def get_user_theme_dir():
"""
Return the Lighthouse user theme directory.
"""
theme_directory = os.path.join(
disassembler.get_disassembler_user_directory(),
"lighthouse_themes"
)
return theme_directory
#----------------------------------------------------------------------
# Properties
#----------------------------------------------------------------------
@property
def TOKEN_COLORS(self):
"""
Return the palette of token colors.
"""
return \
{
# logic operators
"OR": self.logic_token,
"XOR": self.logic_token,
"AND": self.logic_token,
"MINUS": self.logic_token,
# misc
"COMMA": self.comma_token,
"LPAREN": self.paren_token,
"RPAREN": self.paren_token,
#"WS": self.whitepsace_token,
#"UNKNOWN": self.unknown_token,
# coverage
"COVERAGE_TOKEN": self.coverage_token,
}
#----------------------------------------------------------------------
# Callbacks
#----------------------------------------------------------------------
def theme_changed(self, callback):
"""
Subscribe a callback for theme change events.
"""
register_callback(self._theme_changed_callbacks, callback)
def _notify_theme_changed(self):
"""
Notify listeners of a theme change event.
"""
notify_callback(self._theme_changed_callbacks)
#----------------------------------------------------------------------
# Public
#----------------------------------------------------------------------
def warmup(self):
"""
Warms up the theming system prior to initial use.
"""
if self._initialized:
return
logger.debug("Warming up theme subsystem...")
#
# attempt to load the user's preferred (or hinted) theme. if we are
# successful, then there's nothing else to do!
#
self._refresh_theme_hints()
if self._load_preferred_theme():
self._initialized = True
logger.debug(" - warmup complete, using preferred theme!")
return
#
# failed to load the preferred theme... so delete the 'active'
# file (if there is one) and warn the user before falling back
#
try:
os.remove(os.path.join(self.get_user_theme_dir(), ".active_theme"))
except:
pass
disassembler.warning(
"Failed to load Lighthouse user theme!\n\n"
"Please check the console for more information..."
)
#
# if no theme is loaded, we will attempt to detect & load the in-box
# themes based on the user's disassembler theme
#
loaded = self._load_preferred_theme(fallback=True)
if not loaded:
lmsg("Could not load Lighthouse fallback theme!") # this is a bad place to be...
return
logger.debug(" - warmup complete, using hint-recommended theme!")
self._initialized = True
def interactive_change_theme(self):
"""
Open a file dialog and let the user select a new Lighthoue theme.
"""
# create & configure a Qt File Dialog for immediate use
file_dialog = QtWidgets.QFileDialog(
None,
"Open Lighthouse theme file",
self._last_directory,
"JSON Files (*.json)"
)
file_dialog.setFileMode(QtWidgets.QFileDialog.ExistingFile)
# prompt the user with the file dialog, and await filename(s)
filename, _ = file_dialog.getOpenFileName()
if not filename:
return
#
# ensure the user is only trying to load themes from the user theme
# directory as it helps ensure some of our intenal loading logic
#
file_dir = os.path.abspath(os.path.dirname(filename))
user_dir = os.path.abspath(self.get_user_theme_dir())
if file_dir != user_dir:
text = "Please install your Lighthouse theme into the user theme directory:\n\n" + user_dir
disassembler.warning(text)
return
#
# remember the last directory we were in (parsed from a selected file)
# for the next time the user comes to load coverage files
#
if filename:
self._last_directory = os.path.dirname(filename) + os.sep
# log the captured (selected) filenames from the dialog
logger.debug("Captured filename from theme file dialog: '%s'" % filename)
#
# before applying the selected lighthouse theme, we should ensure that
# we know if the user is using a light or dark disassembler theme as
# it may change which colors get used by the lighthouse theme
#
self._refresh_theme_hints()
# if the selected theme fails to load, throw a visible warning
if not self._load_theme(filename):
disassembler.warning(
"Failed to load Lighthouse user theme!\n\n"
"Please check the console for more information..."
)
return
# since everthing looks like it loaded okay, save this as the preferred theme
with open(os.path.join(self.get_user_theme_dir(), ".active_theme"), "w") as f:
f.write(filename)
def refresh_theme(self):
"""
Dynamically compute palette color based on IDA theme.
Depending on if IDA is using a dark or light theme, we *try*
to select colors that will hopefully keep things most readable.
"""
self._refresh_theme_hints()
self._load_preferred_theme()
#--------------------------------------------------------------------------
# Theme Internals
#--------------------------------------------------------------------------
def _populate_user_theme_dir(self):
"""
Create the Lighthouse user theme directory and install default themes.
"""
# create the user theme directory if it does not exist
user_theme_dir = self.get_user_theme_dir()
makedirs(user_theme_dir)
# enumerate all in-box / default themes
plugin_theme_dir = self.get_plugin_theme_dir()
json_files = glob.glob(os.path.join(plugin_theme_dir, "*.json"))
# copy the default themes into the user directory if they don't exist
for default_theme_file in json_files:
theme_name = os.path.basename(default_theme_file)
#
# check if lighthouse has copied the default themes into the user
# theme directory before. when 'default' themes exists, skip them
# rather than overwriting... as the user may have modified it
#
user_theme_file = os.path.join(user_theme_dir, theme_name)
if os.path.exists(user_theme_file):
continue
# copy the in-box themes to the user theme directory
shutil.copy(default_theme_file, user_theme_file)
#
# if the user tries to switch themes, ensure the file dialog will start
# in their user theme directory
#
self._last_directory = user_theme_dir
def _load_required_fields(self):
"""
Load the required theme fields from a donor in-box theme.
"""
logger.debug("Loading required theme fields from disk...")
# load a known-good theme from the plugin's in-box themes
filepath = os.path.join(self.get_plugin_theme_dir(), self._default_themes["dark"])
theme = self._read_theme(filepath)
#
# save all the defined fields in this 'good' theme as a ground truth
# to validate user themes against...
#
self._required_fields = theme["fields"].keys()
def _load_preferred_theme(self, fallback=False):
"""
Load the user's preferred theme, or the one hinted at by the theme subsystem.
"""
logger.debug("Loading preferred theme from disk...")
user_theme_dir = self.get_user_theme_dir()
# attempt te read the name of the user's active / preferred theme name
active_filepath = os.path.join(user_theme_dir, ".active_theme")
try:
theme_name = open(active_filepath).read().strip()
logger.debug(" - Got '%s' from .active_theme" % theme_name)
except (OSError, IOError):
theme_name = None
#
# if the user does not have a preferred theme set yet, we will try to
# pick one for them based on their disassembler UI.
#
if not theme_name:
#
# we have two themes hints which roughly correspond to the tone of
# their disassembly background, and then their general Qt widgets.
#
# if both themes seem to align on style (eg the user is using a
# 'dark' UI), then we will select the appropriate in-box theme
#
if self._user_qt_hint == self._user_disassembly_hint:
theme_name = self._default_themes[self._user_qt_hint]
logger.debug(" - No preferred theme, hints suggest theme '%s'" % theme_name)
#
# the UI hints don't match, so the user is using some ... weird
# mismatched theming in their disassembler. let's just default to
# the 'dark' lighthouse theme as it is more robust
#
else:
theme_name = self._default_themes["dark"]
#
# should the user themes be in a bad state, we can fallback to the
# in-box themes. this should only happen if users malform the default
# themes that have been copied into the user theme directory
#
if fallback:
theme_path = os.path.join(self.get_plugin_theme_dir(), theme_name)
else:
theme_path = os.path.join(self.get_user_theme_dir(), theme_name)
# finally, attempt to load & apply the theme -- return True/False
return self._load_theme(theme_path)
def _validate_theme(self, theme):
"""
Pefrom rudimentary theme validation.
"""
logger.debug(" - Validating theme fields for '%s'..." % theme["name"])
user_fields = theme.get("fields", None)
if not user_fields:
lmsg("Could not find theme 'fields' definition")
return False
# check that all the 'required' fields exist in the given theme
for field in self._required_fields:
if field not in user_fields:
lmsg("Could not find required theme field '%s'" % field)
return False
# theme looks good enough for now...
return True
def _load_theme(self, filepath):
"""
Load and apply the Lighthouse theme at the given filepath.
"""
# attempt to read json theme from disk
try:
theme = self._read_theme(filepath)
# reading file from dsik failed
except OSError:
lmsg("Could not open theme file at '%s'" % filepath)
return False
# JSON decoding failed
except JSONDecodeError as e:
lmsg("Failed to decode theme '%s' to json" % filepath)
lmsg(" - " + str(e))
return False
# do some basic sanity checking on the given theme file
if not self._validate_theme(theme):
return False
# try applying the loaded theme to Lighthouse
try:
self._apply_theme(theme)
except Exception as e:
lmsg("Failed to load Lighthouse user theme\n%s" % e)
return False
# return success
self._notify_theme_changed()
return True
def _read_theme(self, filepath):
"""
Parse the Lighthouse theme file from the given filepath.
"""
logger.debug(" - Reading theme file '%s'..." % filepath)
# attempt to load the theme file contents from disk
raw_theme = open(filepath, "r").read()
# convert the theme file contents to a json object/dict
theme = json.loads(raw_theme)
# all good
return theme
def _apply_theme(self, theme):
"""
Apply the given theme definition to Lighthouse.
"""
logger.debug(" - Applying theme '%s'..." % theme["name"])
colors = theme["colors"]
for field_name, color_entry in theme["fields"].items():
# color has 'light' and 'dark' variants
if isinstance(color_entry, list):
color_name = self._pick_best_color(field_name, color_entry)
# there is only one color defined
else:
color_name = color_entry
# load the color
color_value = colors[color_name]
color = QtGui.QColor(*color_value)
# set theme self.[field_name] = color
setattr(self, field_name, color)
# HACK: IDA uses BBGGRR for its databasse highlighting
if disassembler.NAME == "IDA":
rgb = int(self.coverage_paint.name()[1:], 16)
self.coverage_paint = swap_rgb(rgb)
# all done, save the theme in case we need it later
self.theme = theme
def _pick_best_color(self, field_name, color_entry):
"""
Given a variable color_entry, select the best color based on the theme hints.
"""
assert len(color_entry) == 2, "Malformed color entry, must be (dark, light)"
dark, light = color_entry
# coverage_paint is actually the only field that applies to disas...
if field_name == "coverage_paint":
if self._user_disassembly_hint == "dark":
return dark
else:
return light
# the rest of the fields should be considered 'qt' fields
if self._user_qt_hint == "dark":
return dark
return light
#--------------------------------------------------------------------------
# Theme Inference
#--------------------------------------------------------------------------
def _refresh_theme_hints(self):
"""
Peek at the UI context to infer what kind of theme the user might be using.
"""
self._user_qt_hint = self._qt_theme_hint()
self._user_disassembly_hint = self._disassembly_theme_hint() or "dark"
def _disassembly_theme_hint(self):
"""
Binary hint of the IDA color theme.
This routine returns a best effort hint as to what kind of theme is
in use for the IDA Views (Disas, Hex, HexRays, etc).
Returns 'dark' or 'light' indicating the user's theme
"""
#
# determine whether to use a 'dark' or 'light' paint based on the
# background color of the user's IDA text based windows
#
bg_color = disassembler.get_disassembly_background_color()
if not bg_color:
logger.debug(" - Failed to get hint for disassembly background...")
return None
# return 'dark' or 'light'
return test_color_brightness(bg_color)
def _qt_theme_hint(self):
"""
Binary hint of the Qt color theme.
This routine returns a best effort hint as to what kind of theme the
QtWdigets throughout IDA are using. This is to accomodate for users
who may be using Zyantific's IDASkins plugins (or others) to further
customize IDA's appearance.
Returns 'dark' or 'light' indicating the user's theme
"""
#
# to determine what kind of Qt based theme IDA is using, we create a
# test widget and check the colors put into the palette the widget
# inherits from the application (eg, IDA).
#
test_widget = QtWidgets.QWidget()
#
# in order to 'realize' the palette used to render (draw) the widget,
# it first must be made visible. since we don't want to be popping
# random widgets infront of the user, so we set this attribute such
# that we can silently bake the widget colors.
#
# NOTE/COMPAT: WA_DontShowOnScreen
#
# https://www.riverbankcomputing.com/news/pyqt-56
#
# lmao, don't ask me why they forgot about this attribute from 5.0 - 5.6
#
# IDA 9.2 SDK fix: migrate from PyQt5 to PySide6
# https://docs.hex-rays.com/user-guide/plugins/migrating-pyqt5-code-to-pyside6
if disassembler.NAME == "BINJA" or (disassembler.NAME == "IDA"
and disassembler._version_major == 9
and disassembler._version_minor >= 2):
test_widget.setAttribute(QtCore.Qt.WA_DontShowOnScreen)
else:
test_widget.setAttribute(103) # taken from http://doc.qt.io/qt-5/qt.html
# render the (invisible) widget
test_widget.show()
# now we farm the background color from the qwidget
bg_color = test_widget.palette().color(QtGui.QPalette.Window)
# 'hide' & delete the widget
test_widget.hide()
test_widget.deleteLater()
# return 'dark' or 'light'
return test_color_brightness(bg_color)

Before

Width:  |  Height:  |  Size: 8.7 KiB

After

Width:  |  Height:  |  Size: 8.7 KiB

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 16 KiB

Before

Width:  |  Height:  |  Size: 8.1 KiB

After

Width:  |  Height:  |  Size: 8.1 KiB

Before

Width:  |  Height:  |  Size: 8.1 KiB

After

Width:  |  Height:  |  Size: 8.1 KiB

@@ -0,0 +1,60 @@
{
"name": "Dullien",
"colors":
{
"black": [0, 0, 0],
"white": [255, 255, 255],
"gray": [100, 100, 100],
"lightGray": [220, 220, 220],
"red": [255, 0, 0],
"blue": [0, 0, 255],
"lightRed": [240, 150, 150],
"lightGreen": [150, 240, 150],
"darkGreen": [0, 60, 0],
"lightBlue": [140, 170, 220]
},
"fields":
{
"coverage_paint": ["darkGreen", "lightGreen"],
"table_text": "black",
"table_grid": "gray",
"table_coverage_none": "lightRed",
"table_coverage_bad": "lightRed",
"table_coverage_good": "lightGreen",
"table_background": "white",
"table_selection": "lightBlue",
"html_summary_text": "gray",
"html_table_header": "lightGray",
"html_page_background": "white",
"shell_text": "black",
"shell_text_valid": "blue",
"shell_text_invalid": "red",
"shell_highlight_invalid": "lightRed",
"shell_border": "gray",
"shell_border_focus": "lightBlue",
"shell_background": "white",
"shell_hint_text": "black",
"shell_hint_background": "white",
"logic_token": "red",
"comma_token": "black",
"paren_token": "black",
"coverage_token": "blue",
"combobox_text": "black",
"combobox_selection_text": "white",
"combobox_selection_background": "lightBlue",
"combobox_border": "gray",
"combobox_border_focus": "lightBlue",
"combobox_background": "white"
}
}
@@ -0,0 +1,69 @@
{
"name": "Long Night",
"author": "https://github.com/ioncodes",
"colors":
{
"black": [33, 33, 33],
"white": [241, 239, 236],
"darkGray": [20, 20, 20],
"darkGray2": [30, 30, 30],
"darkGray3": [54, 54, 54],
"gray": [100, 100, 100],
"lightGray": [55, 55, 55],
"red": [188, 101, 141],
"green": [64, 255, 64],
"blue": [104, 134, 197],
"lightBlue": [128, 200, 255],
"darkBlue": [44, 44, 44],
"purple": [121, 104, 197],
"focusRed": [255, 83, 112],
"selection": [67, 67, 67]
},
"fields":
{
"coverage_paint": ["darkBlue", "lightBlue"],
"table_text": "white",
"table_grid": "black",
"table_coverage_none": "black",
"table_coverage_bad": "red",
"table_coverage_good": "blue",
"table_background": "black",
"table_selection": "purple",
"html_summary_text": "white",
"html_table_header": "white",
"html_page_background": "black",
"shell_text": "white",
"shell_text_valid": "lightBlue",
"shell_text_invalid": "red",
"shell_highlight_invalid": "red",
"shell_border": "lightGray",
"shell_border_focus": "focusRed",
"shell_background": "black",
"shell_hint_text": "white",
"shell_hint_background": "black",
"logic_token": "red",
"comma_token": "green",
"paren_token": "green",
"coverage_token": "lightBlue",
"combobox_text": "white",
"combobox_selection_text": "white",
"combobox_selection_background": "selection",
"combobox_border": "lightGray",
"combobox_border_focus": "focusRed",
"combobox_background": "black"
}
}
@@ -0,0 +1,66 @@
{
"name": "Synth",
"colors":
{
"black": [0, 0, 0],
"white": [255, 255, 255],
"darkGray": [20, 20, 20],
"darkGray2": [30, 30, 30],
"darkGray3": [54, 54, 54],
"gray": [100, 100, 100],
"lightGray": [180, 180, 180],
"red": [221, 0, 0],
"green": [64, 255, 64],
"blue": [51, 153, 255],
"lightBlue": [128, 200, 255],
"darkBlue": [0, 0, 153],
"purple": [100, 0, 130]
},
"fields":
{
"coverage_paint": ["darkBlue", "lightBlue"],
"table_text": "white",
"table_grid": "black",
"table_coverage_none": "darkGray2",
"table_coverage_bad": "red",
"table_coverage_good": "blue",
"table_background": "darkGray",
"table_selection": "purple",
"html_summary_text": "lightGray",
"html_table_header": "gray",
"html_page_background": "darkGray3",
"shell_text": "white",
"shell_text_valid": "lightBlue",
"shell_text_invalid": "red",
"shell_highlight_invalid": "red",
"shell_border": "gray",
"shell_border_focus": "lightGray",
"shell_background": "darkGray2",
"shell_hint_text": "white",
"shell_hint_background": "darkGray3",
"logic_token": "red",
"comma_token": "green",
"paren_token": "green",
"coverage_token": "lightBlue",
"combobox_text": "white",
"combobox_selection_text": "white",
"combobox_selection_background": "blue",
"combobox_border": "gray",
"combobox_border_focus": "lightGray",
"combobox_background": "darkGray2"
}
}
@@ -1,3 +1,4 @@
from .python import *
from .misc import *
from .debug import *
from .log import lmsg, logging_started, start_logging
+97
View File
@@ -0,0 +1,97 @@
import sys
import inspect
import cProfile
import traceback
from .log import lmsg
from .disassembler import disassembler
#------------------------------------------------------------------------------
# Debug
#------------------------------------------------------------------------------
#
# This file contains random snippets of code that I frequently use while
# developing and debugging parts of lighthouse. I don't expect any of this
# code to be active or in use for major releases.
#
#------------------------------------------------------------------------------
# Call Profiling
#------------------------------------------------------------------------------
pr = cProfile.Profile()
def profile(func):
"""
A simple function profiling decorator.
"""
def wrap(*args, **kwargs):
global pr
pr.enable()
result = func(*args, **kwargs)
pr.disable()
pr.print_stats(sort="tottime")
return result
return wrap
#------------------------------------------------------------------------------
# Function Line Profiling
#------------------------------------------------------------------------------
try:
import pprofile
def line_profile(func):
def profiled_func(*args, **kwargs):
try:
profiler = pprofile.ThreadProfile()
with profiler():
return func(*args, **kwargs)
finally:
caller_file = inspect.getfile(func)
profiler.annotate(pprofile.EncodeOrReplaceWriter(sys.stdout), [caller_file])
return profiled_func
except ImportError:
def line_profile(func):
def nothing(*args, **kwargs):
return func(*args, **kwargs)
return nothing
#------------------------------------------------------------------------------
# Error Logging
#------------------------------------------------------------------------------
def catch_errors(func):
"""
A simple catch-all decorator to try and log Lighthouse crashes.
This will be used to wrap high-risk or new code, in an effort to catch
and fix bugs without leaving the user in a stuck state.
"""
def wrap(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception:
exc_type, exc_value, exc_traceback = sys.exc_info()
st = traceback.format_stack()[:-1]
ex = traceback.format_exception(exc_type, exc_value, exc_traceback)[2:]
# log full crashing callstack to console
full_error = st + ex
full_error = ''.join(full_error).splitlines()
lmsg("Lighthouse experienced an error... please file an issue on GitHub with this traceback:")
lmsg("")
for line in full_error:
lmsg(line)
# notify the user that a bug occurred
disassembler.warning(
"Something bad happend to Lighthouse :-(\n\n" \
"Please file an issue on GitHub with the traceback from your disassembler console."
)
return wrap
@@ -16,8 +16,9 @@ disassembler = None
if disassembler == None:
try:
from ida_api import IDAAPI, DockableWindow
disassembler = IDAAPI()
from .ida_api import IDACoreAPI, IDAContextAPI
disassembler = IDACoreAPI()
DisassemblerContextAPI = IDAContextAPI
except ImportError:
pass
@@ -27,8 +28,9 @@ if disassembler == None:
if disassembler == None:
try:
from binja_api import BinjaAPI, DockableWindow
disassembler = BinjaAPI()
from .binja_api import BinjaCoreAPI, BinjaContextAPI
disassembler = BinjaCoreAPI()
DisassemblerContextAPI = BinjaContextAPI
except ImportError:
pass
@@ -1,6 +1,9 @@
import abc
import logging
from ..qt import QT_AVAILABLE, QtGui
from ..qt import QT_AVAILABLE, QtGui, QtWidgets
logger = logging.getLogger("Lighthouse.API")
#------------------------------------------------------------------------------
# Disassembler API
@@ -17,9 +20,9 @@ from ..qt import QT_AVAILABLE, QtGui
# to any given interactive disassembler.
#
class DisassemblerAPI(object):
class DisassemblerCoreAPI(object):
"""
An abstract implementation of the required disassembler API.
An abstract implementation of the core disassembler APIs.
"""
__metaclass__ = abc.ABCMeta
@@ -28,41 +31,57 @@ class DisassemblerAPI(object):
@abc.abstractmethod
def __init__(self):
self._waitbox = None
self._ctxs = {}
# required version fields
self._version_major = NotImplemented
self._version_minor = NotImplemented
self._version_patch = NotImplemented
if not self.headless and QT_AVAILABLE:
from ..qt import WaitBox
self._waitbox = WaitBox("Please wait...")
else:
self._waitbox = None
def __delitem__(self, key):
del self._ctxs[key]
def __getitem__(self, key):
return self._ctxs[key]
def __setitem__(self, key, value):
self._ctxs[key] = value
#--------------------------------------------------------------------------
# Properties
#--------------------------------------------------------------------------
@abc.abstractproperty
def version_major(self):
"""
Return the major version number of the disassembler framework.
"""
pass
assert self._version_major != NotImplemented
return self._version_major
@abc.abstractproperty
def version_minor(self):
"""
Return the minor version number of the disassembler framework.
"""
pass
assert self._version_patch != NotImplemented
return self._version_patch
@abc.abstractproperty
def version_minor(self):
def version_patch(self):
"""
Return the patch version number of the disassembler framework.
"""
pass
assert self._version_patch != NotImplemented
return self._version_patch
@abc.abstractproperty
def headless(self):
"""
Return a bool indicating if the disassembler is running headlessly.
Return a bool indicating if the disassembler is running without a GUI.
"""
pass
@@ -97,16 +116,9 @@ class DisassemblerAPI(object):
raise NotImplementedError("execute_ui() has not been implemented")
#--------------------------------------------------------------------------
# API Shims
# Disassembler Universal APIs
#--------------------------------------------------------------------------
@abc.abstractmethod
def get_database_directory(self):
"""
Return the directory for the open database.
"""
pass
@abc.abstractmethod
def get_disassembler_user_directory(self):
"""
@@ -114,6 +126,175 @@ class DisassemblerAPI(object):
"""
pass
@abc.abstractmethod
def get_disassembly_background_color(self):
"""
Return the background color of the disassembly text view.
"""
pass
@abc.abstractmethod
def is_msg_inited(self):
"""
Return a bool if the disassembler output window is initialized.
"""
pass
def warning(self, text):
"""
Display a warning dialog box with the given text.
"""
msgbox = QtWidgets.QMessageBox()
before = msgbox.sizeHint().width()
msgbox.setIcon(QtWidgets.QMessageBox.Critical)
after = msgbox.sizeHint().width()
icon_width = after - before
msgbox.setWindowTitle("Lighthouse Warning")
msgbox.setText(text)
font = msgbox.font()
fm = QtGui.QFontMetricsF(font)
text_width = fm.size(0, text).width()
# don't ask...
spacer = QtWidgets.QSpacerItem(int(text_width*1.1 + icon_width), 0, QtWidgets.QSizePolicy.Minimum, QtWidgets.QSizePolicy.Expanding)
layout = msgbox.layout()
layout.addItem(spacer, layout.rowCount(), 0, 1, layout.columnCount())
msgbox.setLayout(layout)
# show the dialog
msgbox.exec_()
@abc.abstractmethod
def message(self, function_address, new_name):
"""
Print a message to the disassembler console.
"""
pass
#--------------------------------------------------------------------------
# UI APIs
#--------------------------------------------------------------------------
#
# NOTE: please note, these APIs and their usage is a little ... obtuse.
# this is primarily because the IDA & Binja dockable widget management
# system is rather different.
#
# these APIs make a best effort in unifiying the systems in a manner that
# works for this project. it may not be ideal for the universal use case
# but is good enough for our purposes.
#
@abc.abstractmethod
def register_dockable(self, dockable_name, create_widget_callback):
"""
Register a callback with the disassembler to generate dockable widgets.
- dockable_name: the name of the window / dockable to be created
- create_widget_callback: a static function that return a new dockable widget
The registered callback will be called automatically in certain events
that will preclude the display of the dockable_name. These events
may include a new databse being opened, or show_dockable being called.
"""
pass
@abc.abstractmethod
def create_dockable_widget(self, parent, dockable_name):
"""
Creates a dockable widget.
This function should generally be called within the create_widget_callback
described in register_dockable(...).
"""
pass
@abc.abstractmethod
def show_dockable(self, dockable_name):
"""
Show the named dockable widget.
"""
pass
@abc.abstractmethod
def hide_dockable(self, dockable_name):
"""
Hide the named dockable widget.
"""
pass
#------------------------------------------------------------------------------
# WaitBox API
#------------------------------------------------------------------------------
def show_wait_box(self, text, modal=True):
"""
Show the disassembler universal WaitBox.
"""
assert QT_AVAILABLE, "This function can only be used in a Qt runtime"
self._waitbox.set_text(text)
self._waitbox.show(modal)
def hide_wait_box(self):
"""
Hide the disassembler universal WaitBox.
"""
assert QT_AVAILABLE, "This function can only be used in a Qt runtime"
self._waitbox.hide()
def replace_wait_box(self, text):
"""
Replace the text in the disassembler universal WaitBox.
"""
assert QT_AVAILABLE, "This function can only be used in a Qt runtime"
self._waitbox.set_text(text)
#------------------------------------------------------------------------------
# Disassembler Contextual API
#------------------------------------------------------------------------------
class DisassemblerContextAPI(object):
"""
An abstract implementation of database/contextual disassembler APIs.
"""
__metaclass__ = abc.ABCMeta
@abc.abstractmethod
def __init__(self, dctx):
self.dctx = dctx
#--------------------------------------------------------------------------
# Properties
#--------------------------------------------------------------------------
@abc.abstractproperty
def busy(self):
"""
Return a bool indicating if the disassembler is busy / processing.
"""
pass
#--------------------------------------------------------------------------
# API Shims
#--------------------------------------------------------------------------
@abc.abstractmethod
def get_current_address(self):
"""
Return the current cursor address in the open database.
"""
pass
@abc.abstractmethod
def get_database_directory(self):
"""
Return the directory for the open database.
"""
pass
@abc.abstractmethod
def get_function_addresses(self):
"""
@@ -156,12 +337,19 @@ class DisassemblerAPI(object):
pass
@abc.abstractmethod
def navigate(self, address):
def navigate(self, address, function_address=None):
"""
Jump the disassembler UI to the given address.
"""
pass
@abc.abstractmethod
def navigate_to_function(self, function_address, address):
"""
Jump the disassembler UI to the given address, within a function.
"""
pass
@abc.abstractmethod
def set_function_name_at(self, function_address, new_name):
"""
@@ -170,27 +358,13 @@ class DisassemblerAPI(object):
pass
#--------------------------------------------------------------------------
# UI API Shims
# Hooks API
#--------------------------------------------------------------------------
@abc.abstractmethod
def get_disassembly_background_color(self):
def create_rename_hooks(self, function_address, new_name):
"""
Return the background color of the disassembly text view.
"""
pass
@abc.abstractmethod
def is_msg_inited(self):
"""
Return a bool if the disassembler output window is initialized.
"""
pass
@abc.abstractmethod
def warning(self, text):
"""
Display a warning dialog box with the given text.
Returns a hooking object that can capture rename events for this context.
"""
pass
@@ -261,32 +435,6 @@ class DisassemblerAPI(object):
for function_address in function_addresses:
self.clear_prefix(function_address)
#------------------------------------------------------------------------------
# WaitBox API
#------------------------------------------------------------------------------
def show_wait_box(self, text):
"""
Show the disassembler universal WaitBox.
"""
assert QT_AVAILABLE, "This function can only be used in a Qt runtime"
self._waitbox.set_text(text)
self._waitbox.show()
def hide_wait_box(self):
"""
Hide the disassembler universal WaitBox.
"""
assert QT_AVAILABLE, "This function can only be used in a Qt runtime"
self._waitbox.hide()
def replace_wait_box(self, text):
"""
Replace the text in the disassembler universal WaitBox.
"""
assert QT_AVAILABLE, "This function can only be used in a Qt runtime"
self._waitbox.set_text(text)
#------------------------------------------------------------------------------
# Hooking
#------------------------------------------------------------------------------
@@ -316,38 +464,3 @@ class RenameHooks(object):
This will be hooked by Lighthouse at runtime to capture rename events.
"""
pass
#------------------------------------------------------------------------------
# Dockable Window
#------------------------------------------------------------------------------
class DockableShim(object):
"""
A minimal template of the DockableWindow.
this class is only to demonstrate the minimal set of attributes and
functions that a disassembler's DockableWindow class should contain.
show/hide can be overridden entirely depending on your needs, but the
self._widget field should contain a reference to a blank widget that has
been installed into a QDockWidget in the disassembler interface.
"""
__metaclass__ = abc.ABCMeta
def __init__(self, window_title, icon_path):
self._window_title = window_title
self._window_icon = QtGui.QIcon(icon_path)
self._widget = None
def show(self):
"""
Show the dockable widget.
"""
self._widget.show()
def hide(self):
"""
Show the dockable widget.
"""
self._widget.hide()
@@ -0,0 +1,423 @@
# -*- coding: utf-8 -*-
import os
import sys
import logging
import functools
import threading
import collections
from .api import DisassemblerCoreAPI, DisassemblerContextAPI
from ..qt import *
from ..misc import is_mainthread, not_mainthread
import binaryninja
from binaryninja import PythonScriptingInstance, binaryview
from binaryninja.plugin import BackgroundTaskThread
logger = logging.getLogger("Lighthouse.API.Binja")
#------------------------------------------------------------------------------
# Utils
#------------------------------------------------------------------------------
def execute_sync(function):
"""
Synchronize with the disassembler for safe database access.
"""
@functools.wraps(function)
def wrapper(*args, **kwargs):
#
# in Binary Ninja, it is only safe to access the BNDB from a thread
# that is *not* the mainthread. if we appear to already be in a
# background thread of some sort, simply execute the given function
#
if not is_mainthread():
return function(*args, **kwargs)
#
# if we are in the mainthread, we need to schedule a background
# task to perform our database task/function instead
#
# this inline function definition is technically what will execute
# in a database-safe background thread. we use this thunk to
# capture any output the function may want to return to the user.
#
output = [None]
def thunk():
output[0] = function(*args, **kwargs)
return 1
class DatabaseRead(BackgroundTaskThread):
"""
A stub task to safely read from the BNDB.
"""
def __init__(self, text, function):
super(DatabaseRead, self).__init__(text, False)
self._task_to_run = function
def run(self):
self._task_to_run()
self.finish()
# schedule the databases read and wait for its completion
t = DatabaseRead("Accessing database...", thunk)
t.start()
t.join()
# return the output of the synchronized execution / read
return output[0]
return wrapper
#------------------------------------------------------------------------------
# Disassembler API
#------------------------------------------------------------------------------
class BinjaCoreAPI(DisassemblerCoreAPI):
NAME = "BINJA"
def __init__(self):
super(BinjaCoreAPI, self).__init__()
self._init_version()
def _init_version(self):
version_string = binaryninja.core_version()
# retrieve Binja's version #
if "-" in version_string: # dev
disassembler_version = version_string.split("-", 1)[0]
else: # commercial, personal
disassembler_version = version_string.split(" ", 1)[0]
major, minor, patch, *_= disassembler_version.split(".") + ['0']
# save the version number components for later use
self._version_major = major
self._version_minor = minor
self._version_patch = patch
#--------------------------------------------------------------------------
# Properties
#--------------------------------------------------------------------------
@property
def headless(self):
return not(binaryninja.core_ui_enabled())
#--------------------------------------------------------------------------
# Synchronization Decorators
#--------------------------------------------------------------------------
@staticmethod
def execute_read(function):
return execute_sync(function)
@staticmethod
def execute_write(function):
return execute_sync(function)
@staticmethod
def execute_ui(function):
@functools.wraps(function)
def wrapper(*args, **kwargs):
ff = functools.partial(function, *args, **kwargs)
# if we are already in the main (UI) thread, execute now
if is_mainthread():
ff()
return
# schedule the task to run in the main thread
binaryninja.execute_on_main_thread(ff)
return wrapper
#--------------------------------------------------------------------------
# API Shims
#--------------------------------------------------------------------------
def get_disassembler_user_directory(self):
return os.path.split(binaryninja.user_plugin_path())[0]
def get_disassembly_background_color(self):
return binaryninjaui.getThemeColor(binaryninjaui.ThemeColor.LinearDisassemblyBlockColor)
def is_msg_inited(self):
return True
@execute_ui.__func__
def warning(self, text):
super(BinjaCoreAPI, self).warning(text)
def message(self, message):
print(message)
#--------------------------------------------------------------------------
# UI API Shims
#--------------------------------------------------------------------------
def register_dockable(self, dockable_name, create_widget_callback):
dock_handler = DockHandler.getActiveDockHandler()
dock_handler.addDockWidget(dockable_name, create_widget_callback, QtCore.Qt.RightDockWidgetArea, QtCore.Qt.Horizontal, False)
def create_dockable_widget(self, parent, dockable_name):
return DockableWidget(parent, dockable_name)
def show_dockable(self, dockable_name):
dock_handler = DockHandler.getActiveDockHandler()
dock_handler.setVisible(dockable_name, True)
def hide_dockable(self, dockable_name):
dock_handler = DockHandler.getActiveDockHandler()
dock_handler.setVisible(dockable_name, False)
#--------------------------------------------------------------------------
# XXX Binja Specfic Helpers
#--------------------------------------------------------------------------
def binja_get_bv_from_dock(self):
dh = DockHandler.getActiveDockHandler()
if not dh:
return None
vf = dh.getViewFrame()
if not vf:
return None
vi = vf.getCurrentViewInterface()
bv = vi.getData()
return bv
class BinjaContextAPI(DisassemblerContextAPI):
def __init__(self, dctx):
super(BinjaContextAPI, self).__init__(dctx)
self.bv = dctx
#--------------------------------------------------------------------------
# Properties
#--------------------------------------------------------------------------
@property
def busy(self):
return self.bv.analysis_info.state != binaryninja.enums.AnalysisState.IdleState
#--------------------------------------------------------------------------
# API Shims
#--------------------------------------------------------------------------
def get_current_address(self):
# TODO/V35: this doen't work because of the loss of context bug...
#ctx = UIContext.activeContext()
#ah = ctx.contentActionHandler()
#ac = ah.actionContext()
#return ac.address
dh = DockHandler.getActiveDockHandler()
if not dh:
return 0
vf = dh.getViewFrame()
if not vf:
return 0
ac = vf.actionContext()
if not ac:
return 0
return ac.address
@BinjaCoreAPI.execute_read
def get_database_directory(self):
return os.path.dirname(self.bv.file.filename)
@not_mainthread
def get_function_addresses(self):
return [x.start for x in self.bv.functions]
def get_function_name_at(self, address):
func = self.bv.get_function_at(address)
if not func:
return None
return func.symbol.short_name
@BinjaCoreAPI.execute_read
def get_function_raw_name_at(self, address):
func = self.bv.get_function_at(address)
if not func:
return None
return func.name
@not_mainthread
def get_imagebase(self):
return self.bv.start
@not_mainthread
def get_root_filename(self):
return os.path.basename(self.bv.file.original_filename)
def navigate(self, address):
return self.bv.navigate(self.bv.view, address)
def navigate_to_function(self, function_address, address):
#
# attempt a more 'precise' jump, that guarantees to place us within
# the given function. this is necessary when trying to jump to an
# an address/node that is shared between two functions
#
funcs = self.bv.get_functions_containing(address)
if not funcs:
return False
#
# try to find the function that contains our target (address) and has
# a matching function start...
#
for func in funcs:
if func.start == function_address:
break
# no matching function ???
else:
return False
dh = DockHandler.getActiveDockHandler()
vf = dh.getViewFrame()
vi = vf.getCurrentViewInterface()
return vi.navigateToFunction(func, address)
def set_function_name_at(self, function_address, new_name):
func = self.bv.get_function_at(function_address)
if not func:
return
if new_name == "":
new_name = None
state = self.bv.begin_undo_actions()
func.name = new_name
self.bv.commit_undo_actions(state)
#--------------------------------------------------------------------------
# Hooks API
#--------------------------------------------------------------------------
def create_rename_hooks(self):
return RenameHooks(self.bv)
#------------------------------------------------------------------------------
# Function Prefix API
#------------------------------------------------------------------------------
PREFIX_SEPARATOR = "" # Unicode 0x2581
#------------------------------------------------------------------------------
# Hooking
#------------------------------------------------------------------------------
class RenameHooks(binaryview.BinaryDataNotification):
"""
A hooking class to catch symbol changes in Binary Ninja.
"""
def __init__(self, bv):
self._bv = bv
def hook(self):
self._bv.register_notification(self)
def unhook(self):
self._bv.unregister_notification(self)
def symbol_added(self, *args):
self.__symbol_handler(*args)
def symbol_updated(self, *args):
self.__symbol_handler(*args)
def symbol_removed(self, *args):
self.__symbol_handler(*args, True)
def __symbol_handler(self, view, symbol, removed=False):
func = self._bv.get_function_at(symbol.address)
if not func or not func.start == symbol.address:
return
if removed:
self.name_changed(symbol.address, "sub_%x" % symbol.address)
else:
self.name_changed(symbol.address, symbol.name)
def name_changed(self, address, name):
"""
A placeholder callback, which will get hooked / replaced once live.
"""
pass
#------------------------------------------------------------------------------
# UI
#------------------------------------------------------------------------------
if QT_AVAILABLE:
import binaryninjaui
from binaryninjaui import DockHandler, DockContextHandler, UIContext, UIActionHandler
class DockableWidget(QtWidgets.QWidget, DockContextHandler):
"""
A dockable Qt widget for Binary Ninja.
"""
def __init__(self, parent, name):
QtWidgets.QWidget.__init__(self, parent)
DockContextHandler.__init__(self, self, name)
self.actionHandler = UIActionHandler()
self.actionHandler.setupActionHandler(self)
self._active_view = None
self._visible_for_view = collections.defaultdict(lambda: False)
@property
def visible(self):
return self._visible_for_view[self._active_view]
@visible.setter
def visible(self, is_visible):
self._visible_for_view[self._active_view] = is_visible
def shouldBeVisible(self, view_frame):
if not view_frame:
return False
if USING_PYSIDE6:
import shiboken6 as shiboken
else:
import shiboken2 as shiboken
vf_ptr = shiboken.getCppPointer(view_frame)[0]
return self._visible_for_view[vf_ptr]
def notifyVisibilityChanged(self, is_visible):
self.visible = is_visible
def notifyViewChanged(self, view_frame):
if not view_frame:
self._active_view = None
return
if USING_PYSIDE6:
import shiboken6 as shiboken
else:
import shiboken2 as shiboken
self._active_view = shiboken.getCppPointer(view_frame)[0]
if self.visible:
dock_handler = DockHandler.getActiveDockHandler()
dock_handler.setVisible(self.m_name, True)
@@ -1,15 +1,21 @@
import os
import sys
import time
import logging
import binascii
import tempfile
import functools
import idaapi
import idautils
from .api import DisassemblerAPI, DockableShim
if int(idaapi.get_kernel_version()[0]) < 7:
idaapi.warning("Lighthouse has deprecated support for IDA 6, please upgrade.")
raise ImportError
from .api import DisassemblerCoreAPI, DisassemblerContextAPI
from ..qt import *
from ..misc import is_mainthread
from ..misc import is_mainthread, get_string_between
logger = logging.getLogger("Lighthouse.API.IDA")
@@ -48,27 +54,16 @@ def execute_sync(function, sync_type):
return wrapper
#------------------------------------------------------------------------------
# Disassembler API
# Disassembler Core API (universal)
#------------------------------------------------------------------------------
class IDAAPI(DisassemblerAPI):
"""
The IDA implementation of the disassembler API abstraction.
"""
class IDACoreAPI(DisassemblerCoreAPI):
NAME = "IDA"
#
# in IDA 7.0, Hex-Rays refactored the IDA API quite a bit. This
# impacts Lighthouse in a few places, so we use version checks at
# these junctions to determine which API's to use (v7.x or v6.x)
#
# search 'USING_IDA7API' in the codebase for example cases
#
USING_IDA7API = bool(idaapi.IDA_SDK_VERSION >= 700)
def __init__(self):
super(IDAAPI, self).__init__()
super(IDACoreAPI, self).__init__()
self._dockable_factory = {}
self._dockable_widgets = {}
self._init_version()
def _init_version(self):
@@ -86,21 +81,9 @@ class IDAAPI(DisassemblerAPI):
# Properties
#--------------------------------------------------------------------------
@property
def version_major(self):
return self._version_major
@property
def version_minor(self):
return self._version_minor
@property
def version_patch(self):
return self._version_patch
@property
def headless(self):
return False
return idaapi.cvar.batch
#--------------------------------------------------------------------------
# Synchronization Decorators
@@ -122,48 +105,9 @@ class IDAAPI(DisassemblerAPI):
# API Shims
#--------------------------------------------------------------------------
def create_rename_hooks(self):
if self.USING_IDA7API:
class RenameHooks(idaapi.IDB_Hooks):
pass
else:
class RenameHooks(idaapi.IDP_Hooks):
pass
return RenameHooks()
def get_database_directory(self):
return idautils.GetIdbDir()
def get_disassembler_user_directory(self):
return idaapi.get_user_idadir()
def get_function_addresses(self):
return list(idautils.Functions())
def get_function_name_at(self, address):
return idaapi.get_short_name(address)
def get_function_raw_name_at(self, function_address):
if self.USING_IDA7API:
return idaapi.get_name(function_address)
return idaapi.get_true_name(idaapi.BADADDR, function_address)
def get_imagebase(self):
return idaapi.get_imagebase()
def get_root_filename(self):
return idaapi.get_root_filename()
def navigate(self, address):
return idaapi.jumpto(address)
def set_function_name_at(self, function_address, new_name):
idaapi.set_name(function_address, new_name, idaapi.SN_NOWARN)
#--------------------------------------------------------------------------
# UI API Shims
#--------------------------------------------------------------------------
def get_disassembly_background_color(self):
"""
Get the background color of the IDA disassembly view.
@@ -173,31 +117,175 @@ class IDAAPI(DisassemblerAPI):
disassembly view, and take a screenshot of said widget. It will then
attempt to extract the color of a single background pixel (hopefully).
"""
if self.USING_IDA7API:
return self._get_ida_bg_color_ida7()
else:
return self._get_ida_bg_color_ida6()
# method one
color = self._get_ida_bg_color_from_file()
if color:
return color
# method two, fallback
color = self._get_ida_bg_color_from_view()
if not color:
return None
# return the found background color
return color
def is_msg_inited(self):
return idaapi.is_msg_inited()
@execute_ui.__func__
def warning(self, text):
idaapi.warning(text)
super(IDACoreAPI, self).warning(text)
#------------------------------------------------------------------------------
# Function Prefix API
#------------------------------------------------------------------------------
@execute_ui.__func__
def message(self, message):
print(message)
PREFIX_SEPARATOR = "%"
#--------------------------------------------------------------------------
# UI API Shims
#--------------------------------------------------------------------------
def register_dockable(self, dockable_name, create_widget_callback):
self._dockable_factory[dockable_name] = create_widget_callback
def create_dockable_widget(self, parent, dockable_name):
# IDA 9.2 SDK fix: migrate from PyQt5 to PySide6
# https://docs.hex-rays.com/user-guide/plugins/migrating-pyqt5-code-to-pyside6
if USING_PYSIDE6:
from shiboken6 import wrapInstance
else:
import sip
def wrapInstance(ptr, base=None):
return sip.wrapinstance(int(ptr), base)
# create a dockable widget, and save a reference to it for later use
twidget = idaapi.create_empty_widget(dockable_name)
self._dockable_widgets[dockable_name] = twidget
# cast the IDA 'twidget' as a Qt widget for use
widget = wrapInstance(int(twidget), QtWidgets.QWidget)
widget.name = dockable_name
widget.visible = False
# return the dockable QtWidget / container
return widget
def show_dockable(self, dockable_name):
try:
make_dockable = self._dockable_factory[dockable_name]
except KeyError:
return False
parent, dctx = None, None # not used for IDA's integration
widget = make_dockable(dockable_name, parent, dctx)
# get the original twidget, so we can use it with the IDA API's
#twidget = idaapi.TWidget__from_ptrval__(widget) NOTE: IDA 7.2+ only...
twidget = self._dockable_widgets.pop(dockable_name)
if not twidget:
self.warning("Could not open dockable window, because its reference is gone?!?")
return
# show the dockable widget
flags = idaapi.PluginForm.WOPN_TAB | idaapi.PluginForm.WOPN_RESTORE | idaapi.PluginForm.WOPN_PERSIST
idaapi.display_widget(twidget, flags)
widget.visible = True
# attempt to 'dock' the widget in a reasonable location
for target in ["IDA View-A", "Pseudocode-A"]:
dwidget = idaapi.find_widget(target)
if dwidget:
idaapi.set_dock_pos(dockable_name, 'IDA View-A', idaapi.DP_RIGHT)
break
def hide_dockable(self, dockable_name):
pass # TODO/IDA: this should never actually be called by lighthouse right now
#--------------------------------------------------------------------------
# Theme Prediction Helpers (Internal)
#--------------------------------------------------------------------------
def _get_ida_bg_color_ida7(self):
def _get_ida_bg_color_from_file(self):
"""
Get the background color of the IDA disassembly view. (IDA 7+)
Get the background color of the IDA disassembly views via HTML export.
"""
logger.debug("Attempting to get IDA disassembly background color from HTML...")
#
# TODO/IDA: we need better early detection for if IDA is fully ready,
# this isn't effective and this func theme func can crash IDA if
# called too early (eg, during db load...).
#
# this isn't a problem now... but I don't want us to be at risk of
# hard crashing people's IDA in the future should we change something.
#
imagebase = idaapi.get_imagebase()
#if imagebase == idaapi.BADADDR:
# logger.debug(" - No imagebase...")
# return None
# create a temp file that we can write to
handle, path = tempfile.mkstemp()
os.close(handle)
# attempt to generate an 'html' dump of the first 0x20 bytes (instructions)
ida_fd = idaapi.fopenWT(path)
idaapi.gen_file(idaapi.OFILE_LST, ida_fd, imagebase, imagebase+0x20, idaapi.GENFLG_GENHTML)
# IDA 9.x SDK fix: removed `idaapi.eclose`, added `ida_fpro.qfclose`
if int(idaapi.get_kernel_version()[0]) >= 9:
import ida_fpro
ida_fpro.qfclose(ida_fd)
else:
idaapi.eclose(ida_fd)
# read the dumped text
with open(path, "r") as fd:
html = fd.read()
# delete the temp file from disk
try:
os.remove(path)
except OSError:
pass
# attempt to parse the user's disassembly background color from the html (7.0?)
bg_color_text = get_string_between(html, '<body bgcolor="', '">')
if bg_color_text:
logger.debug(" - Extracted bgcolor '%s' from regex!" % bg_color_text)
return QtGui.QColor(bg_color_text)
#
# sometimes the above one isn't present... so try this one (7.1 - 7.4 maybe?)
#
# TODO: IDA 7.5 says c1 is /* line-fg-default */ ... but it's possible c1
# had the bg color of the line in other builds of 7.x? I'm not sure but
# this should be double checked at some point and can maybe just be removed
# in favor of c41 (line-bg-default) as that's what we really want
#
bg_color_text = get_string_between(html, '.c1 \{ background-color: ', ';')
if bg_color_text:
logger.debug(" - Extracted background-color '%s' from line-fg-default!" % bg_color_text)
return QtGui.QColor(bg_color_text)
# -- IDA 7.5 says c41 is /* line-bg-default */, a.k.a the bg color for disassembly text
bg_color_text = get_string_between(html, '.c41 \{ background-color: ', ';')
if bg_color_text:
logger.debug(" - Extracted background-color '%s' from line-bg-default!" % bg_color_text)
return QtGui.QColor(bg_color_text)
logger.debug(" - HTML color regex failed...")
logger.debug(html)
return None
def _get_ida_bg_color_from_view(self):
"""
Get the background color of the IDA disassembly views via widget inspection.
"""
logger.debug("Attempting to get IDA disassembly background color from view...")
names = ["Enums", "Structures"]
names += ["Hex View-%u" % i for i in range(5)]
names += ["IDA View-%c" % chr(ord('A') + i) for i in range(5)]
@@ -208,14 +296,22 @@ class IDAAPI(DisassemblerAPI):
if twidget:
break
else:
raise RuntimeError("Failed to find donor view")
logger.debug(" - Failed to find donor view...")
return None
# touch the target form so we know it is populated
self._touch_ida_window(twidget)
# locate the Qt Widget for a form and take 1px image slice of it
import sip
widget = sip.wrapinstance(long(twidget), QtWidgets.QWidget)
# IDA 9.2 SDK fix: migrate from PyQt5 to PySide6
# https://docs.hex-rays.com/user-guide/plugins/migrating-pyqt5-code-to-pyside6
if USING_PYSIDE6:
from shiboken6 import wrapInstance
else:
import sip
def wrapInstance(ptr, base=None):
return sip.wrapinstance(int(ptr), base)
widget = wrapInstance(int(twidget), QtWidgets.QWidget)
pixmap = widget.grab(QtCore.QRect(0, 10, widget.width(), 1))
# convert the raw pixmap into an image (easier to interface with)
@@ -224,40 +320,6 @@ class IDAAPI(DisassemblerAPI):
# return the predicted background color
return QtGui.QColor(predict_bg_color(image))
def _get_ida_bg_color_ida6(self):
"""
Get the background color of the IDA disassembly view. (IDA 6.x)
"""
names = ["Enums", "Structures"]
names += ["Hex View-%u" % i for i in range(5)]
names += ["IDA View-%c" % chr(ord('A') + i) for i in range(5)]
# find a form (eg, IDA view) to analyze colors from
for window_name in names:
form = idaapi.find_tform(window_name)
if form:
break
else:
raise RuntimeError("Failed to find donor View")
# touch the target form so we know it is populated
self._touch_ida_window(form)
# locate the Qt Widget for a form and take 1px image slice of it
if USING_PYQT5:
widget = idaapi.PluginForm.FormToPyQtWidget(form, sys.modules[__name__])
pixmap = widget.grab(QtCore.QRect(0, 10, widget.width(), 1))
else:
widget = idaapi.PluginForm.FormToPySideWidget(form, sys.modules[__name__])
region = QtCore.QRect(0, 10, widget.width(), 1)
pixmap = QtGui.QPixmap.grabWidget(widget, region)
# convert the raw pixmap into an image (easier to interface with)
image = QtGui.QImage(pixmap.toImage())
# return the predicted background color
return QtGui.QColor(predict_bg_color(image))
def _touch_ida_window(self, target):
"""
Touch a window/widget/form to ensure it gets drawn by IDA.
@@ -274,95 +336,124 @@ class IDAAPI(DisassemblerAPI):
"""
# get the currently active widget/form title (the form itself seems transient...)
if self.USING_IDA7API:
twidget = idaapi.get_current_widget()
title = idaapi.get_widget_title(twidget)
else:
form = idaapi.get_current_tform()
title = idaapi.get_tform_title(form)
twidget = idaapi.get_current_widget()
title = idaapi.get_widget_title(twidget)
# touch/draw the widget by playing musical chairs
if self.USING_IDA7API:
# touch the target window by switching to it
idaapi.activate_widget(target, True)
flush_qt_events()
# touch the target window by switching to it
idaapi.activate_widget(target, True)
flush_qt_events()
# locate our previous selection
previous_twidget = idaapi.find_widget(title)
# locate our previous selection
previous_twidget = idaapi.find_widget(title)
# return us to our previous selection
idaapi.activate_widget(previous_twidget, True)
flush_qt_events()
else:
# touch the target window by switching to it
idaapi.switchto_tform(target, True)
flush_qt_events()
# locate our previous selection
previous_form = idaapi.find_tform(title)
# lookup our original form and switch back to it
idaapi.switchto_tform(previous_form, True)
flush_qt_events()
# return us to our previous selection
idaapi.activate_widget(previous_twidget, True)
flush_qt_events()
#------------------------------------------------------------------------------
# Dockable Window
# Disassembler Context API (database-specific)
#------------------------------------------------------------------------------
class DockableWindow(DockableShim):
"""
A Dockable Qt widget, compatible with IDA 6.8 --> 7.x.
"""
class IDAContextAPI(DisassemblerContextAPI):
def __init__(self, window_title, icon_path):
super(DockableWindow, self).__init__(window_title, icon_path)
def __init__(self, dctx):
super(IDAContextAPI, self).__init__(dctx)
# IDA 7+ Widgets
if IDAAPI.USING_IDA7API:
import sip
self._form = idaapi.create_empty_widget(self._window_title)
self._widget = sip.wrapinstance(long(self._form), QtWidgets.QWidget)
@property
def busy(self):
return not(idaapi.auto_is_ok())
# legacy IDA PluginForm's
else:
self._form = idaapi.create_tform(self._window_title, None)
if USING_PYQT5:
self._widget = idaapi.PluginForm.FormToPyQtWidget(self._form, sys.modules[__name__])
else:
self._widget = idaapi.PluginForm.FormToPySideWidget(self._form, sys.modules[__name__])
#--------------------------------------------------------------------------
# API Shims
#--------------------------------------------------------------------------
# set the window icon
self._widget.setWindowIcon(self._window_icon)
@IDACoreAPI.execute_read
def get_current_address(self):
return idaapi.get_screen_ea()
def show(self):
def get_database_directory(self):
return idautils.GetIdbDir()
def get_function_addresses(self):
return list(idautils.Functions())
def get_function_name_at(self, address):
return idaapi.get_short_name(address)
def get_function_raw_name_at(self, function_address):
return idaapi.get_name(function_address)
def get_imagebase(self):
return idaapi.get_imagebase()
def get_root_filename(self):
return idaapi.get_root_filename()
def navigate(self, address):
return idaapi.jumpto(address)
def navigate_to_function(self, function_address, address):
return self.navigate(address)
def set_function_name_at(self, function_address, new_name):
idaapi.set_name(function_address, new_name, idaapi.SN_NOWARN)
#--------------------------------------------------------------------------
# Hooks API
#--------------------------------------------------------------------------
def create_rename_hooks(self):
return RenameHooks()
#------------------------------------------------------------------------------
# Function Prefix API
#------------------------------------------------------------------------------
PREFIX_SEPARATOR = "%"
#------------------------------------------------------------------------------
# Hooking
#------------------------------------------------------------------------------
class RenameHooks(idaapi.IDB_Hooks):
def renamed(self, address, new_name, local_name):
"""
Show the dockable widget.
Capture all IDA rename events.
"""
# IDA 7+ Widgets
if IDAAPI.USING_IDA7API:
flags = idaapi.PluginForm.WOPN_TAB | \
idaapi.PluginForm.WOPN_MENU | \
idaapi.PluginForm.WOPN_RESTORE | \
idaapi.PluginForm.WOPN_PERSIST
idaapi.display_widget(self._form, flags)
# we should never care about local renames (eg, loc_40804b), ignore
if local_name or new_name.startswith("loc_"):
return 0
# legacy IDA PluginForm's
else:
flags = idaapi.PluginForm.FORM_TAB | \
idaapi.PluginForm.FORM_MENU | \
idaapi.PluginForm.FORM_RESTORE | \
idaapi.PluginForm.FORM_PERSIST | \
0x80 #idaapi.PluginForm.FORM_QWIDGET
idaapi.open_tform(self._form, flags)
rendered_name = idaapi.get_short_name(address)
# call the 'renamed' callback, that will get hooked by a listener
self.name_changed(address, rendered_name)
# must return 0 to keep IDA happy...
return 0
def name_changed(self, address, new_name):
"""
A placeholder callback, which will get hooked / replaced once live.
"""
pass
#------------------------------------------------------------------------------
# HexRays Util
#------------------------------------------------------------------------------
def hexrays_available():
"""
Return True if an IDA decompiler is loaded and available for use.
"""
try:
import ida_hexrays
return ida_hexrays.init_hexrays_plugin()
except ImportError:
return False
def map_line2citem(decompilation_text):
"""
Map decompilation line numbers to citems.
@@ -424,7 +515,7 @@ def map_line2node(cfunc, metadata, line2citem):
# an effort to resolve the set of graph nodes associated with its citems.
#
for line_number, citem_indexes in line2citem.iteritems():
for line_number, citem_indexes in iteritems(line2citem):
nodes = set()
#
@@ -520,4 +611,3 @@ def lex_citem_indexes(line):
# return all the citem indexes extracted from this line of text
return indexes
@@ -2,6 +2,7 @@ import os
import sys
import logging
from .misc import makedirs
from .disassembler import disassembler
#------------------------------------------------------------------------------
@@ -18,7 +19,7 @@ def lmsg(message):
# only print to disassembler if its output window is alive
if disassembler.is_msg_inited():
print prefix_message
disassembler.message(prefix_message)
else:
logger.info(message)
@@ -54,7 +55,8 @@ class LoggerProxy(object):
def write(self, buf):
for line in buf.rstrip().splitlines():
self._logger.log(self._log_level, line.rstrip())
self._stream.write(buf)
if self._stream:
self._stream.write(buf)
def flush(self):
pass
@@ -66,7 +68,7 @@ class LoggerProxy(object):
# Initialize Logging
#------------------------------------------------------------------------------
MAX_LOGS = 5
MAX_LOGS = 10
def cleanup_log_directory(log_directory):
"""
Retain only the last 15 logs.
@@ -80,7 +82,7 @@ def cleanup_log_directory(log_directory):
filetimes[os.path.getmtime(filepath)] = filepath
# get the filetimes and check if there's enough to warrant cleanup
times = filetimes.keys()
times = list(filetimes.keys())
if len(times) < MAX_LOGS:
return
@@ -115,8 +117,11 @@ def start_logging():
# create a directory for lighthouse logs if it does not exist
log_dir = get_log_dir()
if not os.path.exists(log_dir):
os.makedirs(log_dir)
try:
makedirs(log_dir)
except Exception as e:
logger.disabled = True
return logger
# construct the full log path
log_path = os.path.join(log_dir, "lighthouse.%s.log" % os.getpid())
@@ -1,8 +1,16 @@
import os
import re
import errno
import struct
import weakref
import datetime
import threading
import collections
from .python import *
BADADDR = 0xFFFFFFFFFFFFFFFF
#------------------------------------------------------------------------------
# Plugin Util
#------------------------------------------------------------------------------
@@ -48,10 +56,41 @@ def not_mainthread(f):
return f(*args, **kwargs)
return wrapper
#------------------------------------------------------------------------------
# Theme Util
#------------------------------------------------------------------------------
def swap_rgb(i):
"""
Swap RRGGBB (integer) to BBGGRR.
"""
return struct.unpack("<I", struct.pack(">I", i))[0] >> 8
def test_color_brightness(color):
"""
Test the brightness of a color.
"""
if color.lightness() > 255.0/2:
return "light"
else:
return "dark"
#------------------------------------------------------------------------------
# Python Util
#------------------------------------------------------------------------------
def makedirs(path, exists_ok=True):
"""
Make a fully qualified path.
"""
try:
os.makedirs(path)
except OSError as e:
if e.errno != errno.EEXIST:
raise e
if not exists_ok:
raise e
def chunks(l, n):
"""
Yield successive n-sized chunks from a list (l).
@@ -69,6 +108,23 @@ def hex_list(items):
"""
return '[{}]'.format(', '.join('0x%X' % x for x in items))
def human_timestamp(timestamp):
"""
Return a human readable timestamp for a given epoch.
"""
dt = datetime.datetime.fromtimestamp(timestamp)
return dt.strftime("%b %d %Y %H:%M:%S")
def get_string_between(text, before, after):
"""
Get the string between two strings.
"""
pattern = "%s(.*)%s" % (before, after)
result = re.search(pattern, text)
if not result:
return None
return result.group(1)
#------------------------------------------------------------------------------
# Python Callback / Signals
#------------------------------------------------------------------------------
@@ -90,6 +146,7 @@ def register_callback(callback_list, callback):
# 'register' the callback
callback_list.append(callback_ref)
return callback_ref
def notify_callback(callback_list, *args):
"""
@@ -152,92 +209,12 @@ def notify_callback(callback_list, *args):
for callback_ref in cleanup:
callback_list.remove(callback_ref)
#------------------------------------------------------------------------------
# Coverage Util
#------------------------------------------------------------------------------
def coalesce_blocks(blocks):
def unregister_callback(callback_list, callback_ref):
"""
Coalesce a list of (address, size) blocks.
eg:
blocks = [
(4100, 10),
(4200, 100),
(4300, 10),
(4310, 20),
(4400, 10),
]
returns:
coalesced = [(4100, 10), (4200, 130), (4400, 10)]
Remove a previously-registered callback reference.
"""
# nothing to do
if not blocks:
return []
elif len(blocks) == 1:
return blocks
# before we can operate on the blocks, we must ensure they are sorted
blocks = sorted(blocks)
#
# coalesce the list of given blocks
#
coalesced = [blocks.pop(0)]
while blocks:
block_start, block_size = blocks.pop(0)
#
# compute the end address of the current coalescing block. if the
# blocks do not overlap, create a new block to start coalescing from
#
if sum(coalesced[-1]) < block_start:
coalesced.append((block_start, block_size))
continue
#
# the blocks overlap, so update the current coalescing block
#
coalesced[-1] = (coalesced[-1][0], (block_start+block_size) - coalesced[-1][0])
# return the list of coalesced blocks
return coalesced
def rebase_blocks(base, basic_blocks):
"""
Rebase a list of basic block offsets (offset, size) to the given imagebase.
"""
return map(lambda x: (base + x[0], x[1]), basic_blocks)
def build_hitmap(data):
"""
Build a hitmap from the given list of address.
A hitmap is a map of address --> number of executions.
The list of input addresses can be any sort of runtime trace, coverage,
or profiling data that one would like to build a hitmap for.
"""
output = collections.defaultdict(int)
# if there is no input data, simply return an empty hitmap
if not data:
return output
#
# walk through the given list of given addresses and build a
# corresponding hitmap for them
#
for address in data:
output[address] += 1
# return the hitmap
return output
try:
callback_list.remove(callback_ref)
except (ValueError, AttributeError):
pass
+69
View File
@@ -0,0 +1,69 @@
import sys
import operator
#------------------------------------------------------------------------------
# Python 2/3 Compatibilty Shims
#------------------------------------------------------------------------------
PY2 = sys.version_info[0] == 2
PY3 = sys.version_info[0] == 3
#
# xrange shim
#
if PY3:
xrange = range # is this bad lol
#
# Queue --> queue shim
#
try:
import Queue as queue
except:
import queue
#
# iter* shims by Benjamin Peterson, from https://github.com/benjaminp/six
#
if PY3:
def iterkeys(d, **kw):
return iter(d.keys(**kw))
def itervalues(d, **kw):
return iter(d.values(**kw))
def iteritems(d, **kw):
return iter(d.items(**kw))
def iterlists(d, **kw):
return iter(d.lists(**kw))
viewkeys = operator.methodcaller("keys")
viewvalues = operator.methodcaller("values")
viewitems = operator.methodcaller("items")
else:
def iterkeys(d, **kw):
return d.iterkeys(**kw)
def itervalues(d, **kw):
return d.itervalues(**kw)
def iteritems(d, **kw):
return d.iteritems(**kw)
def iterlists(d, **kw):
return d.iterlists(**kw)
viewkeys = operator.methodcaller("viewkeys")
viewvalues = operator.methodcaller("viewvalues")
viewitems = operator.methodcaller("viewitems")
+118
View File
@@ -0,0 +1,118 @@
#
# this global is used to indicate whether Qt bindings for python are present
# and available for use by Lighthouse.
#
QT_AVAILABLE = False
#------------------------------------------------------------------------------
# PyQt5 <--> PySide2 Compatibility
#------------------------------------------------------------------------------
#
# we use this file to shim/re-alias a few Qt API's to ensure compatibility
# between the popular Qt frameworks. these shims serve to reduce the number
# of compatibility checks in the plugin code that consumes them.
#
# this file was critical for retaining compatibility with Qt4 frameworks
# used by IDA 6.8/6.95, but it less important now. support for Qt 4 and
# older versions of IDA (< 7.0) were deprecated in Lighthouse v0.9.0
#
USING_PYQT5 = False
USING_PYSIDE2 = False
USING_PYSIDE6 = False
#
# TODO/QT: This file is getting pretty gross. this whole shim system
# should probably get refactored as I really don't want disassembler
# specific dependencies in here...
#
# IDA 9.2 SDK fix: migrate from PyQt5 to PySide6
# https://docs.hex-rays.com/user-guide/plugins/migrating-pyqt5-code-to-pyside6
try:
import idaapi
ver_major, ver_minor = map(int, idaapi.get_kernel_version().split("."))
USING_NEW_IDA = ver_major == 9 and ver_minor >= 2
USING_OLD_IDA = not(USING_NEW_IDA)
except ImportError:
USING_NEW_IDA = False
USING_OLD_IDA = False
try:
import binaryninjaui
USING_NEW_BINJA = "qt_major_version" in binaryninjaui.__dict__ and binaryninjaui.qt_major_version == 6
USING_OLD_BINJA = not(USING_NEW_BINJA)
except ImportError:
USING_NEW_BINJA = False
USING_OLD_BINJA = False
#------------------------------------------------------------------------------
# PyQt5 Compatibility
#------------------------------------------------------------------------------
# attempt to load PyQt5 (IDA from 7.0 to 9.1)
if USING_OLD_IDA:
try:
import PyQt5.QtGui as QtGui
import PyQt5.QtCore as QtCore
import PyQt5.QtWidgets as QtWidgets
# importing went okay, PyQt5 must be available for use
QT_AVAILABLE = True
USING_PYQT5 = True
# import failed, PyQt5 is not available
except ImportError:
pass
#------------------------------------------------------------------------------
# PySide2 Compatibility
#------------------------------------------------------------------------------
# if PyQt5 did not import, try to load PySide2 (Old Binary Ninja / Cutter)
if not QT_AVAILABLE and USING_OLD_BINJA:
try:
import PySide2.QtGui as QtGui
import PySide2.QtCore as QtCore
import PySide2.QtWidgets as QtWidgets
# alias for less PySide2 <--> PyQt5 shimming
QtCore.pyqtSignal = QtCore.Signal
QtCore.pyqtSlot = QtCore.Slot
# importing went okay, PySide must be available for use
QT_AVAILABLE = True
USING_PYSIDE2 = True
# import failed. No Qt / UI bindings available...
except ImportError:
pass
#------------------------------------------------------------------------------
# PySide6 Compatibility
#------------------------------------------------------------------------------
# If all else fails, try to load PySide6 (New Binary Ninja and IDA)
if not QT_AVAILABLE and (USING_NEW_BINJA or USING_NEW_IDA):
try:
import PySide6.QtGui as QtGui
import PySide6.QtCore as QtCore
import PySide6.QtWidgets as QtWidgets
# alias for less PySide6 <--> PyQt5 shimming
QtCore.pyqtSignal = QtCore.Signal
QtCore.pyqtSlot = QtCore.Slot
QtWidgets.QAction = QtGui.QAction
# importing went okay, PySide must be available for use
QT_AVAILABLE = True
USING_PYSIDE6 = True
# import failed. No Qt / UI bindings available...
except ImportError:
pass
@@ -1,9 +1,11 @@
import sys
import time
import Queue
import logging
import threading
from .shim import *
from ..misc import is_mainthread
from ..python import *
from ..disassembler import disassembler
logger = logging.getLogger("Lighthouse.Qt.Util")
@@ -17,7 +19,7 @@ def MonospaceFont():
Convenience alias for creating a monospace Qt font object.
"""
font = QtGui.QFont("Courier New")
font.setStyleHint(QtGui.QFont.TypeWriter)
font.setStyleHint(QtGui.QFont.Monospace)
return font
#------------------------------------------------------------------------------
@@ -35,8 +37,8 @@ def copy_to_clipboard(data):
Copy the given data (a string) to the system clipboard.
"""
cb = QtWidgets.QApplication.clipboard()
cb.clear(mode=cb.Clipboard)
cb.setText(data, mode=cb.Clipboard)
cb.clear(mode=QtGui.QClipboard.Mode.Clipboard)
cb.setText(data, mode=QtGui.QClipboard.Mode.Clipboard)
def flush_qt_events():
"""
@@ -52,13 +54,6 @@ def get_qt_icon(name):
icon_type = getattr(QtWidgets.QStyle, name)
return QtWidgets.QApplication.style().standardIcon(icon_type)
def get_qt_main_window():
"""
Get the QMainWindow instance for the current Qt runtime.
"""
app = QtCore.QCoreApplication.instance()
return [x for x in app.allWidgets() if x.__class__ is QtWidgets.QMainWindow][0]
def get_default_font_size():
"""
Get the default font size for this QApplication.
@@ -69,8 +64,27 @@ def get_dpi_scale():
"""
Get a DPI-afflicted value useful for consistent UI scaling.
"""
font = QtGui.QFont("Times", 15)
return QtGui.QFontMetricsF(font).xHeight()
font = MonospaceFont()
font.setPointSize(normalize_to_dpi(120))
fm = QtGui.QFontMetricsF(font)
# xHeight is expected to be 40.0 at normal DPI
return fm.height() / 173.0
def compute_color_on_gradient(percent, color1, color2):
"""
Compute the color specified by a percent between two colors.
"""
r1, g1, b1, _ = color1.getRgb()
r2, g2, b2, _ = color2.getRgb()
# compute the new color across the gradient of color1 -> color 2
r = r1 + int(percent * (r2 - r1))
g = g1 + int(percent * (g2 - g1))
b = b1 + int(percent * (b2 - b1))
# return the new color
return QtGui.QColor(r,g,b)
def move_mouse_event(mouse_event, position):
"""
@@ -89,7 +103,9 @@ def normalize_to_dpi(font_size):
"""
Normalize the given font size based on the system DPI.
"""
return (font_size*get_dpi_scale())/5.0
if sys.platform == "darwin": # macos is lame
return font_size + 2
return font_size
def prompt_string(label, title, default=""):
"""
@@ -105,9 +121,12 @@ def prompt_string(label, title, default=""):
dlg.setWindowTitle(title)
dlg.setTextValue(default)
dlg.resize(
dpi_scale*80,
dpi_scale*10
int(dpi_scale*400),
int(dpi_scale*50)
)
dlg.setModal(True)
dlg.show()
dlg.setFocus(QtCore.Qt.PopupFocusReason)
ok = dlg.exec_()
text = str(dlg.textValue())
return (ok, text)
@@ -209,7 +228,7 @@ def await_future(future):
# to the mainthread. flush the requests now and try again
#
except Queue.Empty as e:
except queue.Empty as e:
pass
logger.debug("Awaiting future...")
@@ -268,3 +287,82 @@ def await_lock(lock):
#
raise RuntimeError("Failed to acquire lock after %f seconds!" % timeout)
class QMainthread(QtCore.QObject):
"""
A Qt object whose sole purpose is to execute code on the mainthread.
"""
toMainthread = QtCore.pyqtSignal(object)
toMainthreadFast = QtCore.pyqtSignal(object)
def __init__(self):
super(QMainthread, self).__init__()
# helpers used to ensure thread safety
self._lock = threading.Lock()
self._fast_refs = []
self._result_queue = queue.Queue()
# signals used to communicate with the Qt mainthread
self.toMainthread.connect(self._execute_with_result)
self.toMainthreadFast.connect(self._execute_fast)
#--------------------------------------------------------------------------
# Public
#--------------------------------------------------------------------------
def execute(self, function):
"""
Execute a function on the mainthread and wait for its return value.
This function is safe to call from any thread, at any time.
"""
# if we are already on the mainthread, execute the callable inline
if is_mainthread():
return function()
# execute the callable on the mainthread and wait for it to complete
with self._lock:
self.toMainthread.emit(function)
result = self._result_queue.get()
# return the result of executing on the mainthread
return result
def execute_fast(self, function):
"""
Execute a function on the mainthread without waiting for completion.
"""
#
# append the given function to a reference list.
#
# I do this because I am not confident python / qt will guarantee the
# lifetime of the callable (function) as we cross threads and the
# callee scope/callstack dissolves away from beneath us
#
# this callable will be deleted from the ref list in _excute_fast()
#
self._fast_refs.append(function)
# signal to the mainthread that a new function is ready to execute
self.toMainthreadFast.emit(function)
#--------------------------------------------------------------------------
# Internal
#--------------------------------------------------------------------------
def _execute_with_result(self, function):
try:
self._result_queue.put(function())
except Exception as e:
logger.exception("QMainthread Exception")
self._result_queue.put(None)
def _execute_fast(self, function):
function()
self._fast_refs.remove(function)
qt_mainthread = QMainthread()
@@ -35,6 +35,12 @@ class WaitBox(QtWidgets.QDialog):
qta = QtCore.QCoreApplication.instance()
qta.processEvents()
def show(self, modal=True):
self.setModal(modal)
result = super(WaitBox, self).show()
qta = QtCore.QCoreApplication.instance()
qta.processEvents()
#--------------------------------------------------------------------------
# Initialization - UI
#--------------------------------------------------------------------------
@@ -56,7 +62,7 @@ class WaitBox(QtWidgets.QDialog):
# configure the main widget / form
self.setSizeGripEnabled(False)
self.setModal(True)
self._dpi_scale = get_dpi_scale()
self._dpi_scale = get_dpi_scale()*5
# initialize abort button
self._abort_button = QtWidgets.QPushButton("Cancel")
@@ -77,19 +83,19 @@ class WaitBox(QtWidgets.QDialog):
v_layout.setAlignment(QtCore.Qt.AlignCenter)
v_layout.addWidget(self._text_label)
if self._abort:
self._abort_button.clicked.connect(abort)
self._abort_button.clicked.connect(self._abort)
v_layout.addWidget(self._abort_button)
v_layout.setSpacing(self._dpi_scale*3)
v_layout.setSpacing(int(self._dpi_scale*3))
v_layout.setContentsMargins(
self._dpi_scale*5,
self._dpi_scale,
self._dpi_scale*5,
self._dpi_scale
int(self._dpi_scale*5),
int(self._dpi_scale),
int(self._dpi_scale*5),
int(self._dpi_scale)
)
# scale widget dimensions based on DPI
height = self._dpi_scale * 15
height = int(self._dpi_scale * 15)
self.setMinimumHeight(height)
# compute the dialog layout
+63
View File
@@ -0,0 +1,63 @@
import re
import json
import logging
import threading
try:
from urllib2 import urlopen # Py2
except ImportError:
from urllib.request import urlopen # Py3
logger = logging.getLogger("Lighthouse.Util.Update")
#------------------------------------------------------------------------------
# Update Checking
#------------------------------------------------------------------------------
UPDATE_URL = "https://api.github.com/repos/gaasedelen/lighthouse/releases/latest"
def check_for_update(current_version, callback):
"""
Perform a plugin update check.
"""
update_thread = threading.Thread(
target=async_update_check,
args=(current_version, callback,),
name="Lighthouse UpdateChecker"
)
update_thread.start()
def async_update_check(current_version, callback):
"""
An async worker thread to check for an plugin update.
"""
logger.debug("Checking for update...")
current_version = "v" + current_version
try:
response = urlopen(UPDATE_URL, timeout=5.0)
html = response.read()
info = json.loads(html)
remote_version = info["tag_name"]
except Exception:
logger.debug(" - Failed to reach GitHub for update check...")
return
# convert version #'s to integer for easy compare...
version_remote = int(''.join(re.findall('\d+', remote_version)))
version_local = int(''.join(re.findall('\d+', current_version)))
# no updates available...
logger.debug(" - Local: '%s' vs Remote: '%s'" % (current_version, remote_version))
if version_local >= version_remote:
logger.debug(" - No update needed...")
return
# notify the user if an update is available
update_message = "An update is available for Lighthouse!\n\n" \
" - Latest Version: %s\n" % (remote_version) + \
" - Current Version: %s\n\n" % (current_version) + \
"Please go download the update from GitHub."
callback(update_message)
@@ -16,11 +16,11 @@ if disassembler.headless:
elif disassembler.NAME == "IDA":
logger.info("Selecting IDA loader...")
from lighthouse.ida_loader import *
from lighthouse.integration.ida_loader import *
elif disassembler.NAME == "BINJA":
logger.info("Selecting Binary Ninja loader...")
from lighthouse.binja_loader import *
from lighthouse.integration.binja_loader import *
else:
raise NotImplementedError("DISASSEMBLER-SPECIFIC SHIM MISSING")
Binary file not shown.

Before

Width:  |  Height:  |  Size: 4.3 MiB

After

Width:  |  Height:  |  Size: 2.0 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 204 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 380 KiB

+1558
View File
File diff suppressed because it is too large Load Diff
+14774
View File
File diff suppressed because it is too large Load Diff