Introduction Free Open Access Meducation (FOAM) describes online resources assisting learning in medicine. Little is known about users or their behaviours.
Methods Using Google Analytics for a popular FOAM site (www.paedatricfoam.com), we explored user demographics and patterns of behaviour. We analysed these further with descriptive and statistical tests using SPSS (version 26). Data are presented as mean (SD).
Results There were 181.44 (75.16) mean daily users accessing the site throughout a 4-month period during 2018/2019. 68.9% of users were female; 44% were 25–34 years; 57.3% used a mobile device. The mean session duration was 73.55 (9.41) seconds, with more time spent per session and a greater number of pages per session observed in users accessing the site from a desktop or tablet as opposed to a mobile phone. 84.3% of mobile users left the site after viewing a single page. Referral source was also associated with device used (p<0.001). Age was not related to user behaviours (p>0.05).
Discussion FOAM is a rapidly developing form of medical education, with large user numbers seen for a site just 2 years old. The site is being used by many beyond its intended readership. Rather than accessing multiple pages from a desktop, users have varied online behaviours, with the majority viewing a single page on a mobile phone, referred by social media or Google.
Conclusions Google Analytics can powerfully display usage of medical websites but has important limitations if statistical exploration is required. FOAM users are a heterogenous group, and thus content should be designed with this in mind. Further research must be prioritised focussing on the scope, curriculum coverage, accuracy of information and the effectiveness of FOAM as an educational resource.
- education & training (see medical education & training)
- medical education & training
Statistics from Altmetric.com
Free Open Access Meducation (FOAM) describes the use of social media for medical education.1 More broadly, it describes the community and ethos of information sharing to enhance medical education both using social media and other modalities.1 FOAM modalities include blogs, podcasts, e-texts and websites, as well as resources less conventionally associated with education, such as Twitter and Facebook.2–5
Emergency medicine and critical care specialties have been at the vanguard of FOAM,6 with Cadogan et al 3 identifying over 180 sources of FOAM in 2014, and since then the number has grown substantially. As its name suggests, FOAM is wide-reaching and used worldwide in countries of varying economic status.7 A substantial number of clinicians in developed countries are believed to be using FOAM.8
Research of FOAM has not matched the exponential growth in its use. Only one review of its effectiveness has been published to date.9 While suggesting the potential benefit of FOAM as a learning method, this review highlighted the need for further research. There has also been limited research into the differences in how users interact with different forms of FOAM.9 Unsurprisingly, FOAM users strongly support its use10 and note similarities with self-guided education and internet-based education, where effectiveness has been better demonstrated.11 FOAM can be seen to help educators implement and adapt other evidence-based teaching strategies, such as flipped classroom models12 or journal clubs,13 also allowing users to critically evaluate FOAM resources during their learning.
Currently, published evidence on the use and effectiveness of FOAM use within paediatrics is virtually non-existent. Though recognised within commentaries on the topic,14 15 published peer-reviewed research on FOAM in paediatrics is limited to short-term intervention studies16 and case studies.17 Even in the more popular specialties for FOAM (emergency medicine and critical care), current evidence on user behaviours is typically based around self-report data,8 which present its own issues and associated biases. The limited amount of published data on user behaviour is somewhat surprising, as due to the nature of FOAM, there is great opportunity for collection and evaluation of its user behaviours.8 Carley et al 18 demonstrated how this could be done, publishing data on user engagement on ‘St Emlyn’s Blog’. This highlighted the exceptional growth potential and worldwide reach of FOAM platforms. Two other recent studies have presented similar data in relation to podcast engagement.19 20 Twitter may have even greater potential for analysis, particularly when correlating engagement data with face-to-face events and educational activities.21 22
Currently, very little is known about the users of FOAM and their behaviours when they interact with FOAM sites. We do not know how users find articles, whether this process uses search engines or whether users access FOAM sites similarly to an online newspaper. We do not know who the users are and why they access FOAM rather than other sources of information, nor do we know how this relates to more formal educational or training programmes.
While evidence is starting to accumulate in emergency medicine, this has not been the case for other specialties. We sought to explore patterns of use of one of the two major paediatric FOAM sites, so assisting the development of FOAM and highlighting areas for future research.
Development of Paediatric FOAM (www.paediatricfoam.com) was started in 2016 by trainees and consultants within the London School of Paediatrics. The intention was to promote ‘participation and contribution’, creating an evolving, dynamic, regularly changing section of educational material (www.paediatricfoam.com, 2019).
The site now has 98 articles, written by over 200 contributors. Although not formally peer reviewed, articles are fact checked by a subject matter expert and edited for style. Comments are enabled on all articles, encouraging ongoing informal peer review.
Data collection and analysis
Paediatricfoam.com uses the WordPress platform, allowing recording and evaluation of user data using Google Analytics. Data collected include basic user demographics and location, site behaviours such as duration of time on specific site pages, pages per session and transitions between pages, device used and bounce rate (the degree to which users leave the site after viewing the page they land on, rather than access other pages within the site).
Data were extracted between the dates 11 December 2018 and 30 April 2019 and were exported to Microsoft Excel (version 16.16.18) for basic analysis. Due to the nature of study design, data were presented as mean (SD) or as mean only when SD data were not accessible via the Google Analytics platform. Similarly, the use of statistical tests was limited to variables for which raw data were extractable from the Google Analytics platform.
χ2 tests were performed to evaluate relationships between age and session duration, age and bounce rate, age and average session duration, and between device used and referral source.
A total of 25 583 sessions were logged (mean daily users=181.44, SD=75.16) during the study period. Demographic data were available for 6594 users, of which 4543 (68.9%) were female. Age category distribution is presented in figure 1.
Device data were available for 21 121 users, of which 12 098 (57.3%) accessed the site using mobile devices; 8204 (38.8%) used desktop devices; and 819 (3.8%) used tablet devices. The mean session duration was 73.55s. User age was not associated with mean session duration, bounce rate or pages per visit.
Users accessing the site via mobile devices also viewed fewer pages per session and had lower total session duration than those accessing the site using tablet or desktop (table 1).
Users arrived at the site from a number of sources (figure 2). The majority of users found articles using search engines (53.2%), and 20.9% of users came via social networking sites such as Twitter and Facebook. Referral source was also significantly influenced by device used (p<0.001), for example, with those being referred from social media far more likely to be using mobile devices when compared with those accessing the site via direct search.
The device used to access the site was associated with the pages viewed. Of traffic accessing the site via the most viewed page, 98.2% originated from mobile devices. Those landing on the site’s homepage were mostly using desktop devices (63.44%), whereas of those accessing the page providing information on career development, 50% were using mobile devices and 50% were using desktop devices.
Day of the week influenced user number (figure 3). During weekends, there were fewer daily visitors, although they appeared to have a bounce rate and user behaviours similar to those visiting the site during the middle of the week. Due to limited ability to download raw data for these variables, it was not possible to conduct statistical tests to ascertain whether these differences were statistically significant.
Despite the site and almost all of its contributors being based in the UK, paediatricfoam.com was accessed by users from a total of 146 countries (figure 4).
This study is the first to explore the users of FOAM and their behaviours in detail. It demonstrates the breadth of data and also the limitations of using the Google Analytics website ‘click data’ in research. Although it answers some questions, many more are either raised or left unanswered. When we started this investigation, it was primarily intended to better understand who was reading the articles on the site, how they found them and how they used the site.
We found that even after just 2 years as an active site, it was attracting substantial usage, with a mean daily visitor number of 181.4 (75.2). Little has been reported to compare this to more established journals, although Perneger23 reported an average daily hit rate of 97.8 for articles in the British Medical Journal (impact factor 27.6) in the first week of publication. This usage is less than that reported by Carley et al 18 on their FOAM site, the only other comparable data. There is a plethora of potential reasons for this, ranging from the clinical area, establishment of the site, site design, promotion and search engine optimisation. The effect of these in the context of FOAM is largely unknown.
We saw users were diverse in age, in keeping with several other reports.2 3 5 7 8 24 Of the total users, 68.9% were female, contrasting with the one large-scale study that assessed physician FOAM usage where there was a bias towards the male population,2 although this may also relate to the gender ratios in that clinical specialty. For age, the largest group of paediatric FOAM users was 25–34 years old. Both of these findings are likely to reflect the predominant demographic of paediatric trainees in the UK. McGowan et al 2 and Yoo and Huang25 have suggested that there may be some effect of age and seniority on how individuals used the site, although we found no such relationship.
We saw that mobile phones were the most used device to access the site and that those using them were much more likely to arrive from a search engine. This high proportion may be linked to the largest group of users, known to be active mobile phone users.26 It may also be that the site is being used as a ‘just-in-time’ source of information for questions requiring a quick answer, suggested by the high bounce rate of those using mobile phones on the site. Nakamura27 identified a similar difference in bounce rate in a study of an internet-based education site in a field other than medical education. This suggestion that mobile phones were being used as a just-in-time resource at work might also account for the higher user numbers during weekdays as opposed to weekends.
The findings presented in this paper are all derived from Google Analytics and highlight the limitations of using this tool, as well as its potential advantages. The issues can be divided into those relating to data assurance and the limitations of the accessible information. Although the data are very simple to access from WordPress sites and precisely record usage and trends for the site, it will not be specific in measuring real user behaviour. For instance, Google will aggregate those who finish articles with those that skim the first section only. Gender and age data relate to the owner of the phone, but others may use the phone. Session duration is also not the same as time spent actually reading an article. The challenges of using Google Analytics have been described in other similar studies, and it is advisable to contextualise data with these challenges in mind.28 These deficiencies must be noted, although at the same time other research methods used to evaluate how students or trainees learn have substantial limitations; for instance, attendance at a session is different from active engagement with the learning.
Google Analytics typically presents aggregated data with the raw data not being accessible directly. Mean usage figures for a given time period are readily presented, but understanding the spread around the mean requires sequentially exploring usage for single days of activity. This was required for descriptive and statistical analysis of the data and to explore associations. Analytics also simply reports the use of the site, rather than the use for medical education by clinicians. For instance, there was unusual traffic to the most viewed landing page (umbilical granulomas), featuring 98.2% of traffic from mobile devices. This page was fourth on the Google search results at the time of submission. Parents of children with this condition may be accessing the FOAM site, and they may present a very different type of user to that accessing the rest of the site. Such considerations are a hazard in research on the internet. User behaviours are hard to predict, and a plethora of information and search results available for simple paediatric questions makes defining a user group challenging.29 Additionally, it is not uncommon for parents to access FOAM and directly influence medical decision making as a result.30 As all FOAM is by definition open access, we would expect all FOAM sites to have users beyond the intended group, and excluding these from analysis would create a false view of the site’s performance.
These findings and comments relate to a brief period in the life of a single FOAM site. It may be that other disciplines have different user behaviours, or that when the site is more mature, users will approach it differently.
Conclusions and future research
An understanding of who is accessing FOAM and why they are doing so has not kept pace with its rapid development. We have shown a paediatric site primarily targeting trainees based in London has a broad spread of users across ages and countries, many of whom were accessing the site outside of typical working hours. That said, the majority of users were female, 25–34 years old, and used mobile phones and did so from Monday to Friday, typical demographics and behaviours of paediatric trainees at work.
We learnt that mobile users also outnumbered other user devices and saw that the way users accessed the site was related to behaviour. Desktop and tablet users started with the homepage more often than those on mobiles, who, by contrast, came from social media and search engines. Put together, this gives a different picture of how users may access FOAM from that perhaps existing in site hosts’ minds, that of individuals coming to the site homepage and working through the content. Rather, the majority of users appear to read an individual article, probably only partially, and then leave. However, one must bear in mind the limitations of data collection via Google Analytics when drawing these conclusions.
Perhaps the clearest message from this investigation is around the tool itself. Google Analytics appears to document users and their behaviours, so it can be used as an investigative instrument in its own right. This could extend beyond collecting information about user behaviour to potentially investigating how altering site content or promotion changes activity. However, uncertainties in the validity of the data presented and difficulties in accessing the raw findings limit the tool’s use.
It is also clear that there is a lack of information in this area despite its rapid growth and reach within the medical community. We hope that our findings have made a small contribution to the knowledge of how FOAM is being used. There is a pressing need for further research especially into the effectiveness of FOAM and social media on learning outcomes. With a growing user community, FOAM is already being widely used and in places included in curricula without understanding of best practice or even its benefit to learners. So, choice of FOAM resources may be based more on recommendation and prior experience than any form of validation.
Further investigation could involve
Documenting the prevalence and usage of FOAM across the spectrum of medical specialties and user groups.
Delineating types of FOAM and the way in which these are generated and validated.
Understanding how learners are using FOAM in their professional development or clinical practise.
Exploring what approaches are most useful for learning.
Our investigation has highlighted how FOAM users are a large, heterogenous group, many of whom access content using organic search and social media, and who prefer to use mobile devices rather than desktops. Those curating and creating FOAM resource sites should ensure their articles are easy to find on social media and from search engines, and that they can be easily digested on a phone. As many users are likely to not be medical professionals, appropriate wording is advised. Those researching online medical education will find that while Google Analytics can readily provide data, it is not without its limitations. We suggest that, because of the substantial use of FOAM and its rapid adoption by users, research into the scope, curriculum coverage, accuracy of information and effectiveness is urgently required.
List of learning points
Free Open Access Meducation (FOAM) sites have become an important source of medical information.
Those writing for or curating FOAM should prepare materials so that they are easily accessed from desktop and mobile devices.
Search optimisation is important as a high proportion of traffic is from search engines or social media.
FOAM is global, with a reach far beyond the typical intended audience.
Google Analytics is a powerful and simple-to-use tool for evaluating the use of online educational materials but has its limitations.
Current research questions
Are these findings unique to the paediatric medical education population, or are they generalisable to the wider FOAM community?
What are the key motivators for FOAM users within the paediatric FOAM and wider FOAM communities?
How are users interacting with FOAM, and what are the key features that make an individual source of FOAM appealing to the demographic groups identified?
How can FOAM content contributors optimise their articles/sources to maximise usage and effectiveness within the target population?
To what extent is FOAM now contributing to the learning of postgraduate and undergraduate medical students in comparison to other sources?
We thank the paediatric Free Open Access Meducation community for continuing to contribute and grow this exciting, educational movement.
Contributors All of the stated researchers (JR, KK, JB) were involved in the planning and writing stages of the study. JB was also involved in the data collection and analysis stage. All researchers had oversight and reviewed each stage of the study, via direct communication and progress meetings.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Patient consent for publication Not required.
Ethics approval After discussion with St George’s Research and Ethics Committee, ethical approval was deemed unnecessary for this paper, due to the use of retrospective metadata collected by Google Analytics.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.