From Fedora Project Wiki

< QA‎ | Meetings

Attendees

People present (lines said)

  1. jlaska (93)
  2. kparal (49)
  3. adamw (33)
  4. bmwiedemann (19)
  5. wwoods (10)
  6. Cerlyn (2)
  7. mkrizek (1)
  8. Viking-Ice (1)

Unable to attend:

  1. Rhe (hopefully sleeping)
  2. Hongqing (hopefully sleeping)

Agenda

Previous meeting follow-up

  1. Bodhi feedback patch from fcami (see infrastructure ticket#701 awaiting review
No updates on https://fedorahosted.org/pipermail/bodhi/2010-December/000543.html
  1. adamw rescheduled xorg-x11-drv test events

Update on critpath test definition (ticket#154)

Owner
adamwill
Summary
See fedora-qa ticket#154
Still experimenting with some mock-ups/examples
Next steps ...
HELP - More feedback needed from testers and developers
If no additional feedback, and we assume no news is good news, the next step is converting existing test cases, and getting people started writing new ones developed
Reach out to f-e-k and bodhi teams to discuss tools integration

Latest and greatest on autoqa-0.4.4

Owner
kparal
Summary
Kparal pushed mkrizek's support for staging server into master
Next steps ...
  1. What's the status of depcheck?
    1. Wwoods posting blog article explaining the need for autoqa ticket#248
    2. Determine appropriate strategy for watcher and depcheck integration, and revise patchset as needed
  2. Merge clumens branch into master

SUSE's openqa project

Owner
bmwiedemann
Summary

Open discussion - <Your topic here>

Action items

IRC Transcript

jlaska #startmeeting Fedora QA Meeting 16:00
zodbot Meeting started Mon Jan 10 16:00:21 2011 UTC. The chair is jlaska. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00
zodbot Useful Commands: #action #agreed #halp #info #idea #link #topic. 16:00
jlaska #meetingname fedora-qa 16:00
zodbot The meeting name has been set to 'fedora-qa' 16:00
jlaska #topic Gathering people 16:00
* kparal here 16:01
jlaska okay, show of nicks, who is here? 16:01
* mkrizek is here 16:01
jlaska hi kparal + mkrizek 16:01
jlaska anyone else lurking ... adamw Viking-Ice robatino bmwiedemann? 16:02
adamw yo 16:02
bmwiedemann lurking :) 16:02
* adamw busy playing with wikis 16:02
Cerlyn here 16:02
* Viking-Ice here 16:03
jlaska Cerlyn: adamw: Viking-Ice: bmwiedemann: hello all 16:03
jlaska okay, let's get started 16:03
jlaska quick recap from last week 16:03
jlaska #topic Previous meeting follow-up 16:03
jlaska #info Bodhi feedback patch from fcami (see ticket#701 (infrastructure) awaiting review 16:04
jlaska not much to add here ... looks like it's still pending review 16:04
jlaska I don't see lmacken around to ping on this 16:04
jlaska unless no other ideas ... 16:05
jlaska #info Adamw and Hurry did some f15 test day rescheduling (adjusted Xorg dates and added preupgrade) -- https://fedoraproject.org/wiki/QA/Fedora_15_test_days 16:06
adamw that was in response to viking's suggestion last week 16:06
jlaska right on, thanks Viking-Ice and adamw 16:06
adamw X test week now comes between the first and second GNOME test days 16:06
* jlaska adjusts a cell in the test day schedule 16:07
adamw brb 16:07
jlaska adamw: you're next on the agenda, I'll shuffle that topic until you're bafck 16:07
jlaska back 16:07
jlaska kparal are you ready to talk about autoqa updates? 16:08
kparal jlaska: yes 16:08
jlaska #topic Latest and greatest on autoqa-0.4.4 16:08
jlaska kparal: okay, take it away 16:08
kparal ok, welcome all to the regular news from autoqa world :) 16:08
adamw back 16:08
kparal the changes for the past week: 16:08
jlaska adamw: okay, I'll queue you up right after kparal 16:08
kparal 1. mkrizek's patch "Add support for a staging server" has been pushed to master. 16:09
kparal https://fedorahosted.org/autoqa/ticket/241 16:09
kparal That means that AutoQA should be now fully configurable when it comes to interaction with various external services. Sending email with results (we have 3 different types of them) can be turned on or off, sending Bodhi comments can be turned on or off, URLs for Koji and Bodhi instances are configurable now. There are a few more options available. 16:09
jlaska #info mkrizek's patch "Add support for a staging server" has been pushed to master (https://fedorahosted.org/autoqa/ticket/241) 16:09
kparal 2. my patch "Load config files from current directory by default" has been pushed to master. 16:09
kparal https://fedorahosted.org/autoqa/ticket/253 16:09
kparal All config files are now copied to the autotest client to the test's directory and used from there. That means we don't have to maintain /etc/autoqa conf files on each and every client anymore. 16:09
jlaska very cool, nice work mkrizek and kparal 16:09
kparal Unfortunately in one of the following patches we hit some issues and partly broke this functionality again. But it works for the most important config file - fas.conf. And I'll try to fix the rest soon :) 16:09
jlaska #info my patch "Load config files from current directory by default" has been pushed to master (https://fedorahosted.org/autoqa/ticket/253) 16:09
jlaska kparal: ah, so that's why you wanted site_tests writable by autotest? 16:10
kparal jlaska: yes, that's the reason, all the config files must be first copied to site_tests to be tarred and transfered to the client 16:10
kparal but as I say, maybe we will devise some other solution 16:11
jlaska yeah, I added a comment in the ticket regarding packaging guidelines 16:11
kparal because the current one causes some problems, we can't fully on CWD being set properly on the client 16:11
kparal next on 16:11
kparal 3. clumens asked for merging his clumens branch onto master. His branch expanded again, now it contains not just anaconda_storage test, but also compose_tree and anaconda_checkbot tests and git-post-receive hook. I started to review it, albeit slowly. 16:11
jlaska #info clumens asked for merging his clumens branch onto master (tests: anaconda_storage, compose_tree, anaconda_checkbot, watcher: git-post-receive) 16:12
kparal clumens and jlaska implemented lots of stuff in that patchset, lot of work done in that. so thanks. 16:13
* jlaska notes, it was fun writing test wrappers for existing tests 16:13
kparal 4. jlaska found out that autoqa package will probably need to add autotest as its dependency. (This stuff is still a little unclear, maybe we will need to do a few things differently because of ticket https://fedorahosted.org/autoqa/ticket/256, so this may change). 16:13
jlaska #info jlaska found out that autoqa package will probably need to add autotest as its dependency (due to fas.conf changes) 16:14
kparal everything's interconnected, this is the problem we spoke about a few paragraphs above :) 16:14
kparal 5. A few bugfixes found its way into master branch. Kudos to jskladan for finding them out. 16:14
jlaska #info A few bugfixes found its way into master branch. Kudos to jskladan for finding them out. 16:14
kparal 6. jskladan posted his "New Koji Watcher" patch to autoqa-devel: 16:15
kparal https://fedorahosted.org/pipermail/autoqa-devel/2011-January/001485.html 16:15
kparal That should allow us to get rid of the outdated post-bodhi-update watcher and make use of the new -pending tags in koji. Still waiting for review. 16:15
jlaska #info jskladan posted his "New Koji Watcher" patch to autoqa-devel, ready for review (https://fedorahosted.org/pipermail/autoqa-devel/2011-January/001485.html) 16:15
jlaska kparal: ah okay, I'll queue that up for some feedback today, thanks for reminder 16:15
kparal it will also allow us to run depcheck-style tests effectively, just once for many received update notifications 16:15
kparal 7. Documentation has been improved a little to accommodate the new changes. 16:16
jlaska #info Documentation has been improved a little to accommodate the new changes. 16:16
kparal 8. I just pushed a 'use self.__class__ in super() calls' patch that should simplify a little our test template (one place less requiring manual changes). 16:16
kparal https://fedorahosted.org/pipermail/autoqa-devel/2011-January/001494.html 16:16
jlaska kparal: feel free to prefix with #info if you have more :) 16:16
kparal jlaska: next time, sorry :) 16:16
kparal this is all for today :) 16:17
jlaska #info I just pushed a 'use self.__class__ in super() calls' patch that should simplify a little our test template (https://fedorahosted.org/pipermail/autoqa-devel/2011-January/001494.html) 16:17
jlaska kparal: awesome, thank you for the updates 16:17
jlaska how's it all looking for the next release 16:17
jlaska is the end in sight for the original est date of January? 16:17
kparal we have to solve a few unexpected issues, but I don't see any major obstacle 16:18
kparal I'm just a little worried about testing 16:18
kparal we tested the individual patches manually 16:18
kparal but I can't really say everything's gonna work perfectly when deployed 16:18
jlaska kparal: yeah, it will be interesting to get a sense for how long integration testing takes 16:18
* kparal looking forward to a staging server in the future 16:19
jlaska kparal: note, our new staging hardware has been delivered ... I don't have any updates on ETA for setup 16:19
kparal jlaska: that sounds great 16:19
jlaska kparal: thanks again for the updates 16:20
jlaska adamw: you ready, we can touch on your topic next 16:20
adamw sure 16:20
jlaska #topic Update on critpath test definition (ticket#154) 16:20
jlaska adamw: alrighty, what's the latest n' greatest 16:20
adamw lemme remember, where were we up to last week 16:21
adamw just a sec 16:21
jlaska adamw: https://fedoraproject.org/wiki/QA/Meetings/20110103#Critical_Path_test_case_development 16:21
adamw yeah thanks 16:21
adamw I think the current pages are pretty much good to go, so i'll put them into production soon 16:21
adamw i think the last current easily-resolvable question (ignoring test dependencies for now) is whether we should standardize *naming* 16:22
adamw the benefit of that would be to allow tools like bodhi/f-e-k to display 'nice' test names, but it may be a bit cumbersome 16:22
jlaska adamw: this is for test case page names? 16:23
adamw yes 16:23
adamw so for instance if we standardised on the name QA:Testcase_package_(packagename)_testname 16:23
adamw bodhi could easily weed out all the parameters except 'testname' 16:23
adamw but it may not work well for some situations, i guess. it may be a step too far 16:24
jlaska what's the "standard" now? 16:24
adamw there isn't one 16:24
jlaska the page name has to be unique? 16:24
adamw it has to be QA:Testcase, but then you can do whatever you like 16:24
adamw so far the policy only requires test cases to be in the right categories 16:24
adamw we don't have a de facto standard either, all test cases tend to be named according to somewhat different schemes 16:25
jlaska okay 16:25
adamw (some of them aren't even in the correct QA:Testcase_blah format yet) 16:25
jlaska adamw: yeah, we've got a lot of migration still with older tests 16:26
adamw yeah 16:26
adamw i'm inclined towards not requiring a specific name format as it may restrict some cases 16:26
adamw like creating a test case that can apply as-is to several packages, and adding it to the categories for each package 16:27
jlaska I agree, my take ... let the categories you've previously define handle the organization and metadata ... and let the test case names be more human readable 16:27
adamw and the benefit of being able to display a 'nice' test name isn't that great 16:27
jlaska adamw: that reminds me of one feature of the wiki that I'll need to list on Hurry's feature comparison: human readible test case links 16:28
adamw yeah 16:28
jlaska instead of http://some.server.com/test?id=12435 16:28
jlaska on right, test 12435! that's a good one 16:28
adamw one of my all-time favourites 16:28
jlaska alright ... so any next steps you wanted to highlight? 16:28
adamw i'm migrating some more test cases to the new format at present, still need to move the pages into 'production', announce to the lists, solicit test creation, and talk to tools maintainers 16:29
jlaska #info next steps ... migrating some more test cases to the new format at present, still need to move the pages into 'production', announce to the lists, solicit test creation, and talk to tools maintainers 16:31
jlaska adamw: thanks for the updates. Nice job on the SOP's too 16:31
adamw thanks 16:31
* jlaska adds a comment regarding human-readable test case URLs to https://fedoraproject.org/wiki/Talk:Rhe/tcms_Comparison 16:31
jlaska #topic SUSE openqa project 16:31
jlaska kparal: suggested adding this topic 16:32
jlaska bmwiedemann: joined #fedora-qa last week to let everyone know about a cool test automation effort he started 16:32
jlaska #info Project home - http://openqa.opensuse.org 16:32
jlaska #info Sample automated results available at - http://openqa.opensuse.org/results/ 16:32
jlaska #info Source code hosted at http://www.os-autoinst.org/ 16:32
kparal #info Example Fedora installation video - http://www3.zq1.de/fedora/Fedora-14-i386-netinst.iso.ogv 16:33
jlaska It was nice that bmwiedemann and _lmr_ were both in channel, so they could discuss similarities between the two screenshoting GUI automation approaches (openqa and kvm-autotest) 16:33
adamw ooh! shiny! what's it do? 16:33
kparal adamw: click on the video link 16:33
jlaska adamw: it does GUI test automation by using screen region matching 16:33
jlaska so it's different from LDTP or dogtail, as those are at-spi based 16:33
jlaska I've reached out to hongqing and hurry with some links and for their thoughts on this approach 16:34
jlaska I'll me meeting with _lmr_ after this meeting to talk through his experiences with using GUI matching like this in kvm-autotest 16:34
bmwiedemann The effort started in May 2010 and had produced videos that were also used by openSUSE-marketing on youtube 16:34
kparal bmwiedemann: what is it used for by opensuse - just installation testing, or also desktop applications testing? 16:35
adamw that looks like an awesome step towards project mojito 16:35
jlaska the source is all perl-based ... so if anyone in fedora-qa is a perl guru, this might be fun to get involved in 16:35
bmwiedemann kparal: it does test some important applications like KDE,GNOME, libreoffice, firefox - but the main reason is to prevent breakage in install, boot and updates 16:36
* adamw not a perl guru but he bets he knows where to hire one 16:36
kparal I believe this project could nicely supplement our current rats-install test 16:36
bmwiedemann aka critical-path 16:36
kparal maybe we can execute it from autoqa regularly for Branched composes 16:37
jlaska kparal: it could be ... I've asked hongqing for his thoughts on the approach. He's still coming up to speed with the roadmap 16:37
jlaska bmwiedemann: what do you call your GUI region matches again? 16:38
jlaska kvm-autotest calls them stepfiles 16:38
bmwiedemann I would say "test modules" would be the best equivalent. they contain a "run" method with the keypresses and waits, plus a checklist method with the MD5sums of the regions 16:39
jlaska bmwiedemann: how do you get the md5sums of the regions? Is there a tool that watches a manual install, and records keypresses/clicks etc..? 16:39
kparal bmwiedemann: are you able to ignore minor differences somehow? like changed icon in the menu would not break the test 16:41
bmwiedemann jlaska: every automated install can also take manual interaction via VNC or qemu-monitor commands ("sendkey ctrl-alt-f2"), so I usually interact with the automation, which will write things to a log. 16:41
bmwiedemann but still a lot of things done manually there. 16:41
bmwiedemann kparal: one way to avoid breakage is to only use MD5sum of a certain interesting region. 16:41
kparal bmwiedemann: I see, so I don't have to match the whole screen, just a part of it, right? 16:42
bmwiedemann the alternative is to live with it and have several "known-good" MD5sums for a test 16:42
bmwiedemann kparal: yes. 16:42
jlaska bmwiedemann: I tried running it against a Fedora 14 ISO, but it wasn't happy, and I wasn't sure how/where to make adjustments since I haven't touched perl in ages 16:42
wwoods augh, Perl 16:43
bmwiedemann jlaska: you should know that "wasn't happy" is not a good bug report ;) we can have a look at that later. 16:43
jlaska bmwiedemann: understood :) 16:43
kparal there are some tools in the world that are able to intelligently recognize "almost same" images - they give percentage of similarity. it's done by image downscaling, re-coloring, etc. I am sure some of that approach could be used also here to improve image matching 16:44
jlaska there are some potential licensing things I wasn't sure about with the tool (since it's using some packaging from rpmfusion) to generate the videos. Someone smarter than me would need to work that out 16:44
bmwiedemann kparal: I have one simple approach: thresholding the image before checksumming 16:44
kparal bmwiedemann: how does it work? 16:45
wwoods I'm convinced that for certain apps we could be using a 'translation' that put unique unicode glyphs on each button 16:45
wwoods and then just scan for those glyphs 16:45
bmwiedemann kparal: substituting each byte<128 to 0 and every other to 255 16:45
wwoods heh - we could use tiny QR codes 16:46
jlaska wwoods: would that be any better than just using at-spi instead? 16:46
Cerlyn bmwiedemann: So does your tool support detecting slight shifts of the image region in question (new field, moved for alignment purposes, etc.)? 16:47
bmwiedemann I know, there is "sikuli" out there, which uses opencv for computer-vision to compensate for small changes 16:47
wwoods at-spi requires actual access to the accessibility layer - i.e. the dbus session in the client 16:47
kparal I think both approach have their uses. os-autoinst is very usable for installation testing and boot process testing 16:47
bmwiedemann Cerlyn: no 16:47
jlaska kparal: yeah, I was thinking over the weekend that hte perfect solution would likely be a mix of the two approaches 16:48
kparal Cerlyn: I am sure that can be worked out to some extent 16:48
kparal jlaska: agreed 16:48
wwoods but if we included a special 'translation' of the app, and provided a font to display its 'language', then you can find all the buttons by their labels using image recognition 16:48
jlaska kparal: as we do now with lili's sample DVD auto install proof of concept (uses dogtail + kickstart) 16:48
wwoods so then you could just set anaconda to lang=QR and the button for 'Next' would have the QR-ese word for 'Next', which would be some easily-recognized symbol 16:49
wwoods e.g. a QR code 16:49
kparal bmwiedemann: so what's the tool name - openqa or os-autoinst? :) 16:49
bmwiedemann wwoods: that would only be important, if you wanted to click the buttons, instead of using the keyboard. 16:49
bmwiedemann the test tool is "os-autoinst" 16:49
jlaska wwoods: sounds good, I'd have no idea where+how to start there :) 16:49
bmwiedemann openqa.opensuse.org is just the machine running it 16:50
bmwiedemann one of the two 16:50
jlaska bmwiedemann: thanks for stopping by for the meeting 16:50
wwoods bmwiedemann: true, keyboad will suffice if there's keyboard accelerators for everything you want to test 16:50
wwoods anyway, interesting stuff! 16:50
jlaska all: let's move discussion of further details into #fedora-qa 16:50
jlaska and in the interest of time ... we'll move on to open discussion 16:50
kparal yes, certainly very nice tool 16:50
jlaska agreed, nice work bmwiedemann 16:51
jlaska #topic Open Discussion <your topic here> 16:51
jlaska Alright ... anything folks want to raise that we haven't already discussed? 16:51
jlaska if not ... we'll close out in 2 minutes 16:52
jlaska Meeting end in 1 minute ... 16:53
jlaska thanks everyone for your time! 16:54
jlaska #endmeeting 16:54

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!

Retrieved from "https://fedoraproject.org/w/index.php?title=QA/Meetings/20110110&oldid=214635"