VR Toolkit for Identifying Group Characteristics

Visualising crowds is a key pedestrian dynamics topic, with significant research efforts aiming to improve the current state-of-the-art. Sophisticated visualisation methods are commonly used within modern commercial models, and can improve crowd management techniques and sociological theory development. These models often define standard metrics, including density and speed. However, modern visualisation techniques typically use desktop screens. This can limit the capability of a user to investigate and identify key features, especially in real-world scenarios such as control centres. Virtual reality (VR) provides the opportunity to represent scenarios in a fully immersive environment, granting the user the ability to quickly assess situations. Furthermore, these visualisations are often limited to the simulation model that has generated the dataset, rather than being source-agnostic. This paper presents the implementation of an immersive, interactive toolkit for crowd behaviour analysis. This toolkit was built specifically for use within VR environments and was developed in conjunction with commercial users and researchers. It allows the user to identify locations of interest, as well as individual agents, showing characteristics such as group density, individual (Voronoi) density, speed, and flow. Furthermore, it can be used as a data-extraction tool, building individual fundamental diagrams for all scenario agents, and predicting group status as a function of local agent geometry. Finally, this paper presents an evaluation of the toolkit made by crowd behaviour experts.


Introduction
Crowd simulations have become increasingly important over the last decades in multiple applications (e.g., building evacuation, entertainment and surveillance systems). Crowd analysis software [1] exists for architects and engineers to visually and quantitatively analyse crowd movement datasets to ensure that public spaces and buildings have efficient evacuation plans and enable a smooth flow of pedestrians. Furthermore, crowd analysis is also useful for monitoring interactions between individuals in a crowd [2], helping to advise officials on ways to mitigate the spread of viruses, particularly in dense cities. With so many different use cases for analysing crowds, there is a growing need for applications to visualise this data to make it easy for a user to gain valuable insights.
Crowd data is a type of spatio-temporal data consisting of trajectories for multiple pedestrians. While software [3] and techniques exist to visualise crowds, there are two main limitations. First, these techniques are developed with the goal of displaying data on 2D media like desktop screens. Second, modern crowd analysis software packages have produced high quality visualisation engines, but these are typically limited to displaying the outputs from their own simulation model, rather than being source-agnostic. Virtual Reality (VR) provides the opportunity to mitigate against these limitations, providing a fully immersive visualisation environment, granting users greater spatial perception and unrestricted screen space, which is a fundamental requirement when analysing spatio-temporal data. VR can visualise any generic trajectory dataset, providing a sourceagnostic tool for visualisation.
Visualising spatio-temporal data in a virtual environment using VR also provides new opportunities for the intuitive interaction with data. Natural gestures, such as grabbing, holding and pointing, provide a more interactive and simple method of manipulating data than conventional 2D media. Furthermore, VR stereoscopic technologies also facilitate improved immersion [4] and task involvement [5], two elements that resemble the state of flow described by [6]. Essentially, VR stereoscopic technologies capture the user's whole attention by eliminating the perception of external stimuli and concentrating the user's efforts on the task at hand. VR has also not been extensively studied [7] as a medium for comparative data visualisations. In the context of crowd analysis, it can provide a new method for comparing crowds of people in a way that preserves spatio-temporal relationships between them whilst still making it easy to identify subtle differences in the crowds. In addition to this, VR has been proven to give a better perception of depth compared to desktop screens [8]. Therefore, visualising 4D data (such as the movement of a virtual crowd) in VR allows users to better appreciate all dimensions of the data than if they were viewing it on a desktop screen. In the context of crowds, this improved depth perception can lead to better evaluation of distances between features within the crowd.
Given a specific crowd to analyse, whether it is from collected crowd data or a simulation based on a model, it is beneficial to use a range of methods to gain valuable insights into the crowd. Some common features of crowd analysis include tracking individuals or groups of people in a crowd, as well as analysing statistics about the crowd such as its flow or density. Therefore, this study implemented several tools for crowd analysis:

• Voronoi Density Visualisation Tool
The toolkit provides the option to visualise an animated Voronoi diagram on the environment's floor that changes as the pedestrian simulation progresses. Each polygon represents the pedestrian's local Voronoi cell and each cell's tonality reflects its Voronoi density. The formulation for Voronoi density is provided in Section 2.2.

• Area Analysis Tool
Users can define rectangular areas on the environment's floor, generating a full range of statistics such as the average Voronoi density. The tool can also generate visualisations of these statistics, such as a graph showing the changes in average speed through time, or an animated and interactive density heat map. This density heat map can be used to give a finer-grained view of how density changes within the defined area. The toolkit produces these graphs on individual panels, and allows the user to re-position them anywhere in the scene. Examples of these panels are shown in Figure 5. VR thus provides the opportunity to completely remove the limitation in how much data a user can observe at any time, while simultaneously allowing the user to reorient and choose which data streams to observe. In evacuation planning, this tool can be used to analyse choke points such as doorways or corridors to see if at any point in the simulation these areas are subjected to dangerous levels of density or if the crowd is able to flow smoothly.

• Pedestrian Analysis Tool
Users can select pedestrians in the simulation, creating a subset of the data that only contains the trajectories of those pedestrians. The toolkit generates relevant statistics about the subset such as the minimum, average and maximum speeds and Voronoi densities of each pedestrian at each frame in the simulation. Similar to the area analysis tool, the pedestrian analysis tool can generate visualisations for these statistics, providing the same immersive benefits. In a density analysis task for event planning, this could be used to measure the maximal pedestrian density for all points in the simulation, ensuring it does not exceed a predefined value.

• Social Group Identification
Finally, the toolkit provides the option to identify and examine groups of pedestrians by providing a method to determine group affiliation, and alter the tonality of the pedestrians accordingly. These groups (social or familial) are initially identified based solely on proximity and shared movement direction, however any algorithm can be implemented. The toolkit provides a useful tool for surveillance tasks where the goal is to find a particular group. For such a task, this tool is used to identify all groups in the crowd, changing their tonality so they stand out, making for easier identification. This toolkit was developed to provide functionality to researchers and practitioners, who could then adapt and implement their own required models. As such the algorithm developed by this study was not tested against existing algorithms (such as [9]), or against existing datasets (such as [10]). Future work will develop this toolkit to include state-of-the-art theories surrounding social identification.

Literature Review
This section looks at current methods of analysing crowds, focusing on tracking people within crowds, analysing the density of crowds, and identifying social groups in crowds. First, the section describes methods for tracking and identifying social groups in crowds in terms of their benefits and how they are implemented. Second, this section illustrates both classical and Voronoi density, focusing on the benefits and drawbacks of each method, emphasising the justification for using Voronoi density over classical density. Finally, the visualisation in VR is evaluated in terms of its advantages over regular visualisation on 2D desktop screens.

Tracking and Identifying Social Groups
Tracking pedestrians is crucial for a wide range of applications such as surveillance or evacuation planning. In surveillance, tracking could be used to monitor the actions of a potentially dangerous individual in a crowded scene, and in evacuation planning tracking could be used to analyse the route a pedestrian takes from the top floor of a building to the evacuation exit. In most cases, this sort of analysis is influenced heavily by techniques in computer vision [11] [12], such as segmenting human figures or through clustering people walking close together into 'group tracks', from which individual people can be tracked. These techniques, however, are limited by the resolution and dimensionality of the data, which usually consist of 2D frames in a video that are often taken from poor quality CCTV footage. To represent a 3D scene in 2D, the scene has to be projected into 2D, removing the depth component resulting in occluded details and depth perception is also removed. The benefit of working with data in a 3D spatial context is it avoids the depth ambiguity that comes with 2D images, making it easier for a user to interpret.
Whilst image segmentation techniques can be used for tracking, another approach is to identify pedestrians based on their spatial relationships to other pedestrians. X. Liu [13] illustrates this approach by considering the binary spatial relationships between individuals in a dense crowd and how these are usually preserved over time. An example of a binary spatial relationship between person A and person B might be that A is in front of B, or A is to the left of B. The nature of these relationships through time allow the identification and segmentation of individuals in frames. From these relationships, a probabilistic framework is used to help with tracking pedestrians in 3D scenes. While techniques like this are needed to work with raw data, the data used for the toolkit was labelled, with each pedestrian having a unique identifier, reducing the technical complexity of the tracking algorithm.
Examining pair-wise relationships between individuals can be helpful in tracking individuals as well as identifying groups in crowds. In this case, groups are meant as collections of people such as friends, families or acquaintances walking together in a crowd. Usually, characteristics like gender, age and height help establish whether a group of pedestrians is a family or independent commuters. Humans can often identify these relationships subconsciously [14]. However, this becomes challenging when analysing real-world data. It requires rich datasets consisting of more than just trajectories of pedes-trians, and it is difficult to augment surveillance footage with these characteristics using traditional image-based methods.
Social groups can also be predicted by observing whether pedestrians remain near each other for an extended period of time. The benefit of this method is that it only requires trajectory data, from which the positional and directional relations between individuals can be extracted and used to identify social groups. Yücel et al. [9] approach this by creating positional models based on interpersonal distance and directional models based on relative rotation throughout the entire pedestrians' trajectory to determine the likelihood of each pedestrian being in a group with another. They use this method to identify groups of pedestrians with approximately 85% accuracy.

Density Analysis Methods
When designing a building, the density of a crowd is studied to identify possible locations of bottlenecks which may pose a risk during an emergency evacuation. Classical density is defined as 'people per square metre' [15], and this has been used in planning building developments for analysing pedestrian comfort [16]. The main criticism of classical density is that it does not necessarily give an accurate depiction of density. In reality, classical density suffers from two main issues, outlined by Steffen et al. [17]: • The definition of whether a pedestrian is inside or outside of an area, as they are not a single discrete point. In general, this problem is solved by using the position of the centroid of the pedestrian's head to determine whether they are inside the measurement area.
• Small measurement areas can lead to large spikes in density, and large measurement areas can lead to unrealistically low estimations of density. This is particularly prevalent when the measurement area is only large enough to fit one pedestrian, resulting in the density changing from 0 to maximal density each time a pedestrian enters the area.
In contrast to classical density, Voronoi density [17] uses a Voronoi diagram [18] to find a density distribution for each of the pedestrians. A Voronoi diagram is defined by a set of sites from which every point in the space is assigned to if they are closer to that site than to any other, partitioning a plane into a set of Voronoi cells, one for each site. In the context of density calculations, each site represents a pedestrian's position − → x , where − → x = (x, y) at a particular moment in time and the surrounding cell A i represents the free area surrounding the pedestrian. The density distribution for each pedestrian is then given by the inverse of the cell size |A i |, and the density distribution for all persons is then given by: Following this, the density for a separate measurement area A can be defined as: The above formulation can be implemented by finding which cells intersect with the measurement area and computing a weighted average of the densities of each Voronoi cell, based on how much of the area is taken up by each cell. When compared to classical density, Voronoi density has less scatter and is scale-invariant (i.e., it is not dependent on the measurement area) [17]. The resulting density is intuitive, and takes into account the pedestrian's local conditions.

Visualisation in Virtual Reality
Crowd scenes can be represented as spatio-temporal data. A crowd occupies a space that could be 2D or 3D, and there is the extra dimension of time. Therefore, depending on how the space is defined, the spatio-temporal data for crowds can be considered 3D or 4D. Studies [19] [20] have been conducted on visualising data on screens through 3D rendering. These studies generally show that users find it particularly difficult in judging the size, position and depth of objects, leading to a poorer understanding of the data being shown. These difficulties arise due to the 3D data needing to be flattened in one dimension, depth, in order to display the data on 2D media.
With VR, users can have a more immersive experience as they are no longer restricted to a computer screen but can view data in what appears to be a 3D space with six Degrees of Freedom (DoF). This extra dimension of space leads to users having improved depth perception of 3D objects [21], which in turn improves the user's understanding of the data. Additionally, 3D virtual environments provide further benefits by giving users the complete freedom to create 2D interfaces around themselves in a way that can improve multitasking and analytic reasoning. Examples of this can be seen in Figure 5.

Toolkit Features
The toolkit was developed within a standalone environment developed in Unity, which provided a way for the pedestrian trajectory data to be visualised as pedestrian objects moving through a virtual environment. To develop the toolkit, a variety of features were implemented to help with crowd analysis. The project was developed using the Unity game engine [22] (version 2019.4.0f1) and can be compiled into an Android application package (APK) that can be run on a standalone Oculus Quest head-mounted display (HMD).

Voronoi Density Visualisation
The density visualisations in this toolkit use Voronoi density over classical density as their preferred method of calculation. Figure 1 shows a visualisation of the local density of each pedestrian, by displaying an environment wide Users can extract subsets of the trajectory dataset spatially by defining a rectangular area on the ground. After selecting an area, users can view statistics specific to that area, such as the flow, average speed or Voronoi density in that area, as well as visualisations of these statistics. For this study, speed of a pedestrian was determined as the magnitude of the difference in position over the previous 1 second. The average velocity of a subset of pedestrians at a particular time was calculated using the total number of pedestrians in the subset, and the sum of their instantaneous speeds. Finally, the flow was calculated using by multiplying the average velocity by the total number of pedestrians. Figure 2 shows the features that are available for analysing area subsets. Users can see how statistics (Panel 1) such as the number of people or Voronoi density change in that area in real-time as the simulation runs.
The toolkit also visualises various 2D relationships in the data (Panel 2) such as how the speed or (Voronoi) density change with. Each graph comes with an additional side panel, displaying relevant aggregated statistics throughout the course of the simulation. Currently, the toolkit only supports speed-time and density-time visualisations for area subsets. However, the graphing functionality has been designed to work with any form of 2D data. This means that it is easy to extend the tool to support additional relationships such as flow-density and speed-flow, with the user only having to provide the 2D data that they wish to visualise. To see how the density changes within the area subset throughout the simulation, users can view an interactive and animated Voronoi density heat map (Panel 3) for the area subset. They are also able to specify the time interval of the simulation that they wish to investigate. The Voronoi density for the area subset is found by first seeing which Voronoi cells intersect with the area subset. The overall density is then calculated by averaging the densities of the intersecting cells, weighted by the area of their intersection with the defined area subset. The density heat map as seen in Figure 3 splits the area measurement into square cells with a width of 0.5m. The method for calculating Voronoi density for an area subset is then applied to each square cell to find a Voronoi density for that cell. For efficiency, only the Voronoi cells that intersected with the original area subset are checked to see if they intersect with the cell, rather than all Voronoi cells in the environment.
Pedestrian Analysis Tool Subsets of the trajectory data can also be created by selecting groups of pedestrians, with no restrictions on how many pedestrians can be selected.
Similarly to the area analysis tool, Figure 4 shows the various data panels available to the user to aid with analysing the pedestrian subset. Users can see real-time statistics for the subset as the simulation plays. As each pedestrian has their own Voronoi density, the density-time graph for pedestrian subsets plots the minimum, average, and maximum densities, aggregated for the pedestrians in the subset, over the course of the simulation. A fundamental diagram is also available to the user wishing to examine speed-density relationships. This is computed for the pedestrian subset, plotting the average speed of the pedestrian at the observed range of densities.

Social Group Identification
The social group identification tool as seen in Figure 5, provides an easy-to-understand visualisation of the estimated social groups in the crowd. These social groups are precomputed from the dataset by considering entire pedestrian trajectories and whether multiple trajectories are similar (i.e., if their interpersonal distance and relative rotation are lower than a certain threshold throughout the duration of the simulation). A limit to the number of people that can exist in a single social group was set at 5, although this can be varied by the user. If a social group exists, the pedestrians are highlighted in that group in a unique colour, to enable to user to distinguish between different social groups and non-group pedestrians. The toolkit was designed to be as mod-  Currently the toolkit uses Algorithm 1 which shows how two pedestrians are grouped, initially assuming none of the pedestrians are in groups. The algorithm iterates through each pedestrian, checking if they form a social group with any of their neighbours. This is done by finding the square of the interpersonal distance between both pedestrians and their relative rotations. If they are less than the predefined maximum distance (3 metres) and rotation (15 • ), then a value corresponding to the number of times this condition has been satisfied is incremented. If the two pedestrians are classified as a group for the majority of their trajectories (in this instance, at a minimum of 2 seconds less than the duration of their entire trajectory), then they are considered to exist in the same social group. The neighbours of the pedestrian are found by checking at the edges that make up that pedestrian's cell in the Voronoi diagram. A slightly different algorithm exists for the case where the given pedestrian is already in a group. If this is the case, the interpersonal distance is compared to the group's centre, and the relative rotation is compared to everyone in the group.

Toolkit Performance and Scalability
The aim for this toolkit is to be as widely available and usable as possible. As such, this project investigated the requirements for listing the toolkit on the Oculus Store. For an app to be listed on the Oculus Store, Oculus has created a set of performance targets that an app should meet before it can be released. To be applicable, the baselines for the app are the following: • 72 FPS (frames per second) required • 150-175 maximum draw calls per frame • 750,000 -1,000,000 maximum triangles per frame.
The toolkit meets the targets for draw calls and triangles per frame, however, it does not meet the recommended FPS. For typical usage of the app, which includes running the pedestrian simulation and using each of the toolkit features, the toolkit achieves an average FPS of 60. The most performance-intensive component of the VR app is rendering the pedestrian simulation, which the toolkit has been built on top of.
In terms of the toolkit features, the most performance-intensive feature is the area subset selection tool, which can cause the toolkit to momentarily freeze if the defined area is particularly large. The main cause of this is due to the heat map generation, which is computationally expensive. This was optimised in Unity by utilising Unity's Job System and the Burst compiler. Unity's Job System [23] is Unity's approach to multi-threading, which ensures that the created threads do not interfere with Unity's main thread. When using the Job System, the developer can also enable the Burst compiler [24] which is a special 'math-aware' compiler that can produce highly optimised native code. The combination of the Job System and the Burst compiler brought the time to create a heat map for a 30m x 20m area down from 18s to 0.8s.
To evaluate the scalability of the area subset tool, the changes in FPS were observed after the generation of area subsets of increasing size, from 30 m 2 to 600 m 2 . Figure 6 shows the average FPS of the app, during which four area subsets were created with areas: 1 -30 m 2 , 2 -60 m 2 , 3 -300 m 2 , 4 -600 m 2 . The performance impact for creating the first three subsets is not large, but for 4 a considerable amount of time is spent creating the subset, causing the app to freeze. This limitation can be removed by pre-calculating the heat maps.

Expert study
An expert study was carried out to understand the potential applications and usability of this toolkit. Six industry professionals and researchers were contacted and shown videos of the use of this toolkit, before being asked to respond to a template survey. They pro- vided their responses to several questions surrounding the applicability of the toolkit for modern uses, as well as any desired areas of improvement. The questions asked the participants to rate the toolkit using a 5-point Likert scale (1: Not at all useful, 3: Moderately useful, 5: Very useful) in its ability to perform work in: • Research • Industry

• Visualisation
The roles of the participants and their initial scores are detailed in Table 1. Further comments, and suggested uses are detailed in Table 2.
The output from the expert study shows that there is a large potential demand for this type of toolkit. However, there were mixed responses, with no clear indication as to a pattern. For example, the scores for the toolkit's impact on visualisation ranged from 2 to 5, while the scores for the its impact on research varied only from 3 to 3.5. However, this variation may be a result of the fact that the toolkit was shown using 2D videos, rather than using an HMD headset or other VR equipment. This was done as a result of logistical and time limitations due to the COVID-19 pandemic, and may have limited the potential to showcase the opportunities of the toolkit.
There was also feedback suggesting improvements, such as incorporating Fruin LOS (Level Of Service) [25] categories, or the potential to reduce the complexity of the tool. Several responses revolved around the use of this toolkit for visualising of different conditions such as variable densities for large scale infrastructure, as well as the potential for its use in providing commercial clients a better experience using real-world data.

Conclusion and future work
This paper has detailed the development and evaluation of a VR toolkit that focuses on visualising and analysing the movement of pedestrian. This toolkit was developed to provide an environment in which the user can investigate aspects of crowd motion such as density, speed, or social groups. This approach builds on previous research that visualises the output of simulation models (often as part of the simulation toolkit itself), and The option to overlay graphs within the software itself is a new feature and may be useful for live monitoring or reproduction of live environments (i.e., prediction of crowd flows using real time data). From an analysis perspective, I have seen similar types of analyses from other commercial software, albeit the production of these simulations is likely to take longer than a Unity based product. N/A combines it with the potential for visualising real-world datasets, in an immersive and intuitive manner. This is particularly relevant given the modern requirements for crowd density management, as well as the increases in the VR commercial market size.
The toolkit was shown to industry and academic experts, who provided feedback on its implementation and potential usage. This showed mixed results, with participants identifying high potential for the toolkit to be used commercially, while also commenting on the need for it to be low complexity. The authors welcome any feedback and proposals for collaborations for use of this toolkit, with a future aim to provide the environment as an open source resource, and with further possible extension into AR visualisations.

Acknowledgment
This work has been supported by H2020 EU project RISE, grant agreement No 739578.