-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Increase CPython test coverage #1671
Comments
@palaviv is it possible to generate and online dashboard that shows which tests pass and which need more help? |
I am sure it is possible... Would you like to create such a dashboard? I think you can use testgrid for that. |
@palaviv is it possible to get the report in a file in machine readable format, such as https://testanything.org ? |
The test use |
Would it help having the dashboard somewhere on the current website? If you are looking for a dashboard, we can do two pages:
The goal would be for people to know what they can help on in one place. I can probably get the output of both scripts to push to github repos and then use that as part of jekyll page or just as something people can read. |
I think that would be cool, and it would probably require getting the test results in a machine readable format like TAP. It seems like tappy works perfectly fine on RustPython, the only issue is getting regrtest to run a custom unittest test runner. |
@mireille-raad is it be possible to have single dashboard for both passed/failed and "what is left" by color coding them with green, red and yellow? |
I don't know if it makes sense to have them in the same dashboard, since the regrtests are tests that get run and are well structured, while the whats_left script just shows standard library items that aren't present in RustPython; they don't really show the same kind of data. |
@coolreader18 |
Well yeah, sort of; all whats_left.sh does is diff the list of standard library modules and the |
@coolreader18 but do testnames match between what is available in RustPython and upstream? If they do, then it might be possible to get the test results from both test servers in TAP format and diff them. |
That's not really necessary, since every test that's in CPython is passing (or should be, at least), so we can assume that any test that fails on RustPython should be passing, no diffing required. |
|
@coolreader18 there should be platform dependent tests that are skipped in CPython. Therefore the system configuration of test environment for CPython and RustPython should be the same. |
Ah, I see what you mean. My idea for that was to check if the skip message contains the string "RUSTPYTHON", since so far (just by convention) all the tests that we've marked as |
@abitrolly do you think you can figure out how to generate one or a couple of machine readable files for the test results? (my preference would be json, but i can manage if xml or just plain text)... it looks like you have a specific idea of what need to be done. i can then put them together on a page. |
I've created a PR for JSON output for CPython tests, #1834, you just need to run |
yay... i will take a look at the output, do a sketch. share here. get your feedback on gitter. if things look ok, we should be ready to roll soon! |
I am excited to see how the JSON output looks like as well. Need to figure out how to provide it as a part of CI build result. |
@coolreader18 can you automate running the tests and putting the .json file in the other repo https://github.com/RustPython/rustpython.github.io? under the _data/ directory? |
Yeah, I think that would be possible, maybe running it once a week would be good? |
@mireille-raad I've set up a cron job in #1866, and already tested it a few times, so the results file is already pushed to the git repo. |
thank you! I saw it yesterday night and already created a draft page... I'll use the other repo for discussion on design, page layout and such. |
@coolreader18 very nice. ) It would be easier to review changes in test coverage if the produced |
@abitrolly for readiablity, you can always format the document in your code editor or soon enough, go to the web page on rustpython.github.io ... adding indentation and spacing will make the file even bigger. usually you compress json files, not inflate them. |
Maybe https://rustpython.github.io/pages/regression-tests-results.html output can be improved to include not-implemented tests as a separate color. |
Thanks! I will look at the guides and try to fix some :) |
part of: RustPython#1671
part of: RustPython#1671
part of: RustPython#1671
part of: RustPython#1671
part of: RustPython#1671
part of: RustPython#1671
part of: RustPython#1671
part of: RustPython#1671
part of: RustPython#1671
part of: RustPython#1671
part of: RustPython#1671
part of: RustPython#1671
I removed |
@youknowone can't this be closed in favor of #4564? They both overlap but 4564 is more organized and up to date. |
I removed adding test from the issue body but kept fixing errors. does it make sense? We may can open new one with more organized form using dashboard |
We are currently running part of the CPython test suite. This is a good place for people new to the project to contribute. Contribution here can come in two ways:
Add more tests suites - Copy test suite from CPython and mark failing tests with xfail or skip.In order to run the test suite use
cargo run -- -m test -v
.All tests are in
Lib/test
Guide: https://rustpython.github.io/blog/2020/04/04/how-to-contribute-by-cpython-unittest.html
The text was updated successfully, but these errors were encountered: